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ABSTRACT 

This study examined whether the Chicago Annenberg Challenge 
promoted improvement in schools it supported and in student achievement and 
other outcomes; factors that might explain improvement or lack thereof among 
Annenberg schools; and what could be learned from the Challenge’s 
experiences. It highlighted the period between 1996-1997 through 2000-2001. 
Results suggest that among the schools it supported, the Challenge had little 
impact on school improvement and student outcomes, with no statistically 
significant differences between Annenberg and non-Annenberg schools in rates 
of achievement gain, classroom behavior, student self-efficacy, and social 
competence. The study also examined trends in school improvement among a 
small group of "Breakthrough Schools," which received special financial and 
professional support from the Challenge between 1990-2001, a time during 
which the Challenge began withdrawing funds from the other Annenberg schools. 
Overall, Breakthrough schools began to develop in ways that distinguished 
them from other Annenberg schools and sustained or strengthened aspects of 
teacher professional community, school leadership, and relational trust while 
other Annenberg schools did not. Factors distinguishing strong and weak 
schools included cultivation of strong, distributive leadership and use of an 
array of complementary, reinforcing strategies. Appendixes contain research 
methodology and results. (Contains 59 figures, 27 tables, and 131 endnotes.) 
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Executive Summary 

This final technical report of the Chicago Annenberg Research Project 
addresses four central questions: (a) Did the Chicago Annenberg 
Challenge promote improvement of the schools that it supported? (b) 
Among those schools, did it also promote improvement in student 
academic achievement and other outcomes? (c) What factors might 
explain improvement or lack thereof among Annenberg schools? and (d) 
What can we learn from the Challenge’s experiences to promote school 
improvement in the future? In answer to these questions, this report 
provides a macro view of the Challenge’s success in promoting school 
improvement and student learning. Additionally, it looks closely at 
several Annenberg schools to understand what makes local school 
improvement successful. 



This report focuses on the period between the 1996 to 1997 and 2000 to 2001 
school years, the five full years during which the Challenge supported local school 
improvement. In all, the Challenge supported about 210 high schools and 
elementary schools, but because approximately 90 percent of these were elementary 
schools, this report focuses only on them. 

The Challenge’s “bottom line” was improving student achievement and other 
social and psychological outcomes. Our research indicates that student outcomes in 
Annenberg schools were much like those in demographically similar non-Annenberg 
schools and across the Chicago school system as a whole, indicating that among the 
schools it supported, the Challenge had little impact on student outcomes. 
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• Analyses of ITBS scores indicate that between 1996 and 2001, student academic 
achievement improved across Annenberg schools as it did across the Chicago 
Public School system as a whole. At the same time, rates of gain in student 
achievement among Annenberg schools did not improve markedly. There were 
no statistically significant differences between Annenberg schools and non- 
Annenberg schools in rates of achievement gain. 

• Across Annenberg schools, student academic engagement was only slightly 
greater in 2001 than before the Challenge. Classroom behavior, students’ sense of 
self-efficacy, and social competence were weaker in 2001 than before the 
Challenge. Like student academic achievement, there were no statistically 
significant differences in these outcomes between Annenberg schools and non- 
Annenberg schools. 



Although Annenberg schools did not achieve an overall effect on student 
outcomes, we examined whether the Challenge promoted improvements in schools 
that might lead to subsequent improvement in student outcomes. Using the Model 
of Essential Supports for Student Learning as a framework for analysis, we assessed 
seven areas of school improvement: (a) quality of classroom instruction; (b) student 
learning climate; (c) school leadership; (d) teacher professional community; (e) 
parent and community support; (f) relational trust; and (g) instructional program 
coherence. 

Findings from large-scale survey analyses, longitudinal field research, and student 
achievement test score analyses reveal that while the Challenge contributed to the 
improvement of a number of Annenberg schools, there is little evidence of an overall 
Annenberg school improvement “effect.” Any improvements were much like those 
occurring in demographically similar non-Annenberg schools. 



• The overall quality of instruction improved somewhat among Annenberg 

schools, particularly teachers’ use of interactive teaching strategies, the intellectual 
demand of instruction, and teachers’ emphasis on writing. Some aspects of 
student learning climate also improved, particularly school safety and classroom 
personalism. Some small improvements occurred in school leadership, teacher 
professional community, parent involvement in schools, and relational trust. At 
the same time, other areas failed to improve and some weakened. These included 
student peer support for academic learning, inclusive school leadership, and 
teacher commitment to school. 

via 
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• Initial improvement that occurred by 1 999 among Annenberg schools in a 
number of areas of school organizational capacity — school leadership, teacher 
professional community and professional development, parent and community 
support, relational trust, and instruction program coherence disappeared by the 
end of the Challenge in 2001. Although some measures of organizational 
capacity were slightly stronger or weaker in 2001 than at the beginning of the 
Challenge, there was little net change. In all, the organizational capacity of 
Annenberg schools at the end of the Challenge looked much like it did at the 
beginning. 

Factors that might explain the lack of an overall Annenberg effect on school 
improvement and student outcomes include (a) various shortcomings in the design 
and implementation of the Challenge, including broad goals and vague strategies, too 
few resources for too many schools, and weak levers for change; (b) External Partners’ 
lack of capacity; (c) schools’ lack of capacity to “do Annenberg,” including 
weaknesses in human, social, and material resources; (d) schools’ lack of commitment 
to the Challenge; (e) sources of disruption and persistence within schools; and (f) 
countervailing forces outside of Annenberg schools, notably the school system’s high- 
stakes accountability policies. The loss of initial improvement among Annenberg 
schools may have resulted from both the decline in Annenberg financial and 
professional support after 1999, and intensified CPS accountability policies. 

In addition to trends in school improvement across all Annenberg schools, this 
report examined trends in school improvement among a small group of 
“Breakthrough Schools.” These Breakthrough Schools received special financial and 
professional support from the Challenge between 1999 and 2001. At the same time, 

the Challenge began withdrawing funding from the remainder of Annenberg schools. 

/ 

The findings indicate that Breakthrough Schools began to develop in ways that 
distinguished them from other Annenberg schools. Although there were no 
statistically significant differences between Breakthrough Schools and other 
Annenberg schools in 1999, the year they were selected by the Challenge, 
Breakthrough Schools sustained or strengthened aspects of teacher professional 
community, school leadership, and relational trust while other Annenberg schools 

IX 
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did not. This suggests that these schools may have built a stronger foundation for 
subsequent development of instruction and student learning climate, and this, in 
turn, may promote future improvement in student outcomes. 

Factors that might explain the relative success of Breakthrough Schools include 
(a) greater initial capacity for development, coupled with (b) different and sustained 
resources, and perhaps (c) a motivational boost from their selection. 

The fourth component of our research helped us to better understand what 
factors influenced successes and failures in Annenberg schools. We examined the 
development of 12 Annenberg schools over a five-year period and studied the 
relationship of their development to the improvement activities in which they 
engaged. Four factors emerged from this study that distinguished strong schools and 
schools that improved from those that were weak and those that did not improve. 
Strong schools or schools that improved focused on improving multiple, mutually- 
reinforcing aspects of school organization and practice (e.g., classroom instruction 
and aspects of school leadership and professional community that might support 
instructional improvement) rather than a single aspect of school organization or 
practice (e.g., instructional improvement alone). They used an array of 
complementary, reinforcing strategies (e.g., professional development with incentives 
and accountability) rather than only one (e.g., accountability or professional 
development alone). Schools that were strong or that improved were generally more 
effective at searching for, securing, and taking full advantage of external resources. 
However, what distinguished these schools from nonimproving schools was the 
ability to secure resources aligned with a particular development agenda and to 
employ these resources in an efficient and strategic manner. Finally, schools that were 
strong or that improved distinguished themselves from weak schools and 
nonimproving schools by the cultivation of strong, distributive leadership. Teacher 
leaders make substantial contributions to school improvement, but this analysis 
highlights the “make-or-break” role of the principal, even when people in different 
roles join principals in “leadership work.” 
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This report concludes with several lessons drawn from the experience of the 
Chicago Annenberg Challenge for promoting future large-scale school improvement. 
First, while it may be important to encourage local pluralism and self-determinism in 
developing, adopting, and implementing initiatives to improve schools, it may be 
equally important to provide guidance for local initiatives in the form of well- 
researched and well-thought out maps of change. Second, it may be more effective to 
concentrate greater amounts of resources on a relatively small number of schools that 
are selected in part for their capacity to implement the particular reform at hand. 

This report argues it is less effective to distribute relatively small amounts of resources 
among a very large number of schools that have been selected with less 
discrimination. Third, adequate and sustained Financial support for school 
improvement is essential, but as important is how that money is spent. Money 
appears to be a necessary but insufficient resource to promote and support school 
improvement. Also important are intellectual, social, and political resources that 
build upon and extend a schoofs existing resources. Resources should be aligned with 
coherent goals and plans for school improvement. Fourth, constructive interaction 
with and engagement of the school system seems to be an important ingredient for 
supporting local school improvement. Conflicts and contradictions among reform 
initiatives and system policies pose implementation problems at the school and 
classroom levels. Finally, school improvement is a difficult and complex task that 
requires hard and sustained work over long periods of time. While it may be foolish 
to spend too much time and too many resources on bad reform ideas, it is also 
foolish to give up prematurely on potentially effective ones. 
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Part One: Introduction 



In 1995, the Chicago Annenberg Challenge launched a six-year, large- 
scale initiative to improve Chicago’s public schools. It set out a broad 
vision for change, calling for the “enhancement of learning for all 
students through dramatically improved classroom practice and 
strengthened community relationships.” 1 The Challenge funded 
networks of schools and External Partners to plan, develop, and 
implement activities to improve local schools and student learning. At its 
peak, it supported improvement activities in about 210 schools in the 
Chicago public school system. These focused on many different areas of 
school organization and practice, including curriculum and instruction, 
student learning climate and social services, teacher and leadership 
development, and the involvement of parents and the community in 
schools and student learning. 



Overview 

This final technical report of the Chicago Annenberg Research Project describes the 
Chicago Annenberg Challenge and the broader context of Chicago school reform 
within which it was established and operated. It documents changes among 
participating Annenberg schools from 1996 to 1997 through 2000 to 2001, the five 
full school years the Challenge supported local school improvement activity. It also 
presents trends among Annenberg schools in academic achievement and other 
student outcomes during the same period. The report analyzes the strengths and 
weaknesses of the Challenge as a strategy for promoting large-scale local school 
improvement and identifies a number of factors that may have affected what it was 
able to accomplish. Finally, drawing on the experiences of the Chicago Challenge, it 
discusses several lessons about how to promote urban school improvement in the 



1 Hallett, Chapman, and Ayers (1995). See also Sconzert, Shipps, and Smylie (1998). 
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future. This report follows and extends two previous technical reports, Getting 
Started: A First Look at Chicago Annenberg Schools and Networks and Development of 
Chicago Annenberg Schools , 1996-1 999. 2 

This report addresses four central questions: (a) Did the Chicago Annenberg 
Challenge promote improvement of the schools it supported? (b) Among those 
schools, did it also promote improvement in student achievement and nonacademic 
outcomes? (c) What factors might explain improvement or the lack thereof among 
Annenberg schools? and (d) What can we learn from the Challenge’s experience to 
promote school improvement in the future? In the process of answering these ques- 
tions, this report provides a general assessment of the overall success of the Chicago 
Challenge in promoting school improvement and student learning. 

This report focuses primarily on the Challenge as a whole and on the large group 
of schools it supported. The vast majority of these — about 90 percent — were 
elementary schools. Because so few high schools participated in the Challenge, and 
because of the unevenness of data available on them, we only discuss elementary 
schools here. We draw upon citywide survey and student achievement data to 
identify trends in school change and student outcomes across Annenberg schools. As 
described in our discussion of research methodology, a significant part of the 
Chicago Annenberg Research Project was longitudinal field research. We draw on 
this field research to illustrate broad trends across Annenberg schools in survey data. 
Although much can be learned about large-scale school improvement from 
examining Annenberg schools as a whole, much can also be learned by looking at the 
experiences of individual schools. So, this report contains a section that draws on the 
field research to examine closely differences between improving and nonimproving 
schools and to understand in-depth what helps make local school improvement 
successful. 

Overall, this report presents a story of a particular large-scale, decentralized 
approach to educational reform. It is a complex story from which we can draw 
important lessons about how to make reform more efficient and effective. 



2 Smylie et al. (1998) and Wenzel et al. (2001). The Challenge made its first grants to networks and 
External Partners in December 1995. Winter and spring of 1996 were used primarily for planning 
and development. For most funded networks, implementation of development activities did not begin 
in earnest until the fall of the 1996-97 school year. 
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The Chicago Annenberg Challenge 

The Chicago Annenberg Challenge was established in January 1995 with a $49.2 
million grant from the Annenberg Foundation. 3 It was one of six such projects that 
received funding that year or the year before. Other Challenges were established in 
New York, Philadelphia, Los Angeles, and the San Francisco Bay area. A national 
network of rural schools also received a grant. Since 1995, additional projects were 
begun in other cities. As a condition of funding, all projects were required to raise a 
two-to-one match of additional money or in-kind contributions. In addition, each 
had to commission and support its own local evaluation. The Consortium on Chicago 
School Research conducted the Chicago study referred to as the Chicago Annenberg 
Research Project. 

The Chicago Challenge grew out of the city’s 1988 school decentralization 
reform, which shifted substantial authority for local school governance from the 
Chicago Public Schools (CPS) central administration to local school communities. 
The Challenge was based on the premise that taking reform beyond school 
governance meant allowing teachers, parents, and communities to rethink and 
restructure public schools. The Challenge reflected a particular view of democratic 
localism and community organizing that placed great faith in the ability of local 
schools, in partnership with parents and their communities, to define their own 
problems, challenge their own assumptions, identify their own goals for 
improvement, and develop their own strategies to achieve them. This was very 
different from “old reform” that prescribed goals and strategies and attempted to 
improve schools from the “top down.” The Challenge eschewed establishing 
common goals and designating particular programs or strategies to achieve them. It 
followed the principles of the national Annenberg Challenge, a group associated with 
the Annenberg Institute that coordinated, monitored, and supported the work of the 
different local Challenges. The Annenberg Institute described these principles in 
1998 as follows: 

An abiding tenet of the Annenberg Challenge since its inception, one 
that distinguishes it from other major school reform initiatives, is its 
embrace of pluralism. Believing that there is no magic bullet, no 
single panacea, for fixing what ails our nation's most troubled 
schools, the Challenge has eschewed privileging one reform strategy 
over another. Rather, like all pluralist efforts, the Challenge 
accommodates an array of theories, in this case about how change 
occurs in schools and in the systems of which they are a part. And 



3 For a more detailed description of the establishment and early organizational history of the 
Challenge, see Shipps and Sconzert with Swyers (1999). 
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like all pluralist efforts, its constituent elements are characterized by 
both similarities and differences. 4 

Consistent with these principles, the Chicago Challenge was organized and 
operated much like a foundation that provided financial support to a large number 
of locally defined and developed improvement initiatives. In addition to money, 
the Challenge sought to promote local school improvement by connecting grant 
recipients to schools with similar problems and interests and to External Partners in 
networks of mutual support and assistance. Although the Challenge provided 
school leaders and External Partners some guidance for developing their funding 
proposals and hosted some professional development and workshop activities, it 
believed that the primary source of support for local improvement activity would 
come from the relationships among the schools and Partners that worked together. 
It did not develop a strong central program of technical assistance. 

Goals of the Challenge 

The goals of the Chicago Challenge were broad and diffuse and evolved over the 
course of the initiative. The proposal that brought the 1995 grant from the 
Annenberg Foundation laid out the overall goal of the Chicago Challenge this way: 

The goal of the Annenberg Challenge in Chicago is to increase 
student learning and achievement in Chicago schools. The Challenge 
will be the catalyst for a dramatic increase in [the] renewal of active 
and effective instruction, classroom change, and school 
reorganization at a significant number of schools. 

The proposal laid out a number of outcomes that would be achieved. It claimed that 
participating schools would be “dramatically restructured” with respect to the size of 
student enrollment, time for student learning, and teacher professional development. 
Teachers would become leaders in developing curricular and instructional 
innovation. Local School Councils (LSCs), the school-level parent and community- 
based governing bodies established through the 1988 reform, would grow in their 
knowledge of effective practices. As a result, student academic achievement would 
improve and students’ social and emotional development would be enhanced. 
Overall, the Challenge would further energize the 1988 reform movement in 
Chicago. 

Initially, the Challenge sought to focus local school improvement activity by 
encouraging its grantees to address several basic problems of school organization. 
“Time, size, and isolation” were seen as impediments to improving teaching and 



4 Annenberg Institute for School Reform (1998). 
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student learning. These organizational problems are discussed below. Midway 
through its work (at the same time the school system was drawing attention to 
student academic achievement and performance on standardized tests), the Challenge 
reasserted its initial goals of improving instruction and student learning. It also 
encouraged “whole school change,” that is, schoolwide improvement rather than 
improvement aimed at only individuals or small groups within schools. 

In addition, the Chicago Challenge sought to influence the course of school 
reform in the city. This goal was promoted by the national Annenberg Challenge 
across all the local Challenges and was embraced by local organizers of the Chicago 
Challenge, most of whom were school reform advocates and community organizers 
who had been involved with the development of the 1988 school decentralization 
reform. 



Strategy for Promoting School Improvement 

The Challenge intended to build upon Chicago's 1988 decentralization reform and 
extend the changes that were achieved in school-level governance to other areas of 
school improvement and student learning. Its primary strategy was to create networks 
of schools with common interests and needs and to link them to individuals and 
organizations that would serve as External Partners. This strategy followed a logic 
that schools would find more direction and support for improvement if they worked 
together and with an External Partner than if they worked alone. Partners were to 
perform a number of different functions. They were to serve as fiduciary agents of 
Annenberg grants. They were to bring human, material, intellectual, and occasionally 
political resources to support local school improvement. They were to create focus 
and sustain imperative to develop local leadership and help schools in their networks 
support each other. Partners were also encouraged by the Challenge to bring 
additional financial resources to local school improvement efforts. Initially, the 
Challenge saw networks as the main agents for local school improvement. Over time, 
as networks struggled to develop, External Partners became more and more central to 
the Challenge's strategy. 5 

As noted above, the Chicago Challenge did not articulate specific goals for 
individual school development, nor did it specify any particular activities or processes 
to follow. Rather, it believed that educators, parents, and community members could 
and should identify their own ways to solve local problems and improve their 
schools. The Challenge initially encouraged schools to focus their efforts on 
addressing three basic problems of school organization that were seen as obstacles to 



5 For more information about Annenberg External Partners and their work, see Newmann and 
Sconzert (2000) and Sconzert, Wenzel, and Smylie (2003). 
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improvement: (a) the lack of time for effective teaching, student learning, and teacher 
professional development; (b) the large size of school enrollments and instructional 
groups hindering the development of personalized, supportive adult-student 
relationships; and (c) schools’ isolation from parents and communities, which 
reduced their responsiveness to local needs and their accountability to their most 
immediate constituents. Isolation was later extended to include teachers’ isolation 
from one another, which could limit opportunities for teacher learning and 
development, innovation, and professional accountability. 

In making its first network grants, the Challenge encouraged schools to address 
one or more of these organizational problems. Thereafter, it encouraged schools and 
External Partners to focus more specifically on teaching and student academic 
learning, teacher professional development, and whole school change. 6 Schools and 
External Partners that received funding were asked to demonstrate how their 
Annenberg-supported activities might lead to improvement in student learning. 

Later, the Challenge accepted grant applications by invitation only and did not renew 
the funding of several particularly weak networks. In its last two years, the 
Challenge concentrated a substantial amount of its remaining resources on a group 
of selected “Breakthrough Schools.” The Breakthrough School initiative is 
described later in this section. 



Breadth and Depth of Support 

The Chicago Challenge made two types of grants. It distributed small amounts of 
money in one-year planning grants to schools and External Partners to develop 
networks and school improvement plans. Additionally, schools and Partners could 
apply for implementation grants that provided larger sums of money to support 
school improvement activity. Implementation grants usually supported several years 
of activity and could be renewed. Due to the large number of schools that received 
implementation grants, and because of their amount relative to planning grants, this 
research project focused only on schools that were members of networks receiving 
implementation grants. 

Beginning in 1995, the Challenge made implementation grants to 45 External 
Partners and their networks of schools. The size of networks ranged from three to 
15 schools. The average network consisted of four to five schools. Through the 
networks, the Challenge funded a large number of elementary, middle, and high 



6 These developments are examined in more detail in other reports of the Chicago Annenberg 
Research Project. See Newmann and Sconzert (2000); Shipps and Sconzert with Swyers (1999); and 
Smylie et al. (1998). 
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schools — as many as 21 1, or about 40 percent of all schools in the Chicago public 
school system. Approximately 90 percent were elementary schools. The Challenge 
awarded most of its implementation grants in two major waves. Thirty-four 
networks received initial funding at the end of 1995; the remaining networks first 
received funding in 1997. 7 The total number of schools receiving funds rose from 
138 in 1996 to 21 1 in 1998. From 1999 through 2001 , the last year of school and 
network funding, the Challenge supported about 206 schools (see Figure 1). 



Figure 1. Number of Schools in Chicago Annenberg Implementation Networks, 1996 to 2001 




1996 1997 1998 1999 2000 2001 



Overall, Annenberg schools resembled schools across the system. As a group, 
those schools that received grants in the first wave of funding differed somewhat 
from the system as a whole in that their enrollments were slightly larger and had 
somewhat larger proportions of low-income and low-achieving students. Also, a 
slightly larger proportion of Annenberg schools than schools citywide had 
enrollments that were more than 85 percent African-American or more than 85 
percent African-American and Latino. By the 1998-99 school year, as a result of the 
inclusion of schools funded in the second wave, these differences all but disappeared. 
The average enrollment size, level of academic achievement, racial and ethnic 
composition, and percentage of low-income students (those eligible for federal free 
and reduced-price lunch programs) in Annenberg elementary schools were virtually 
identical to the system as a whole (see Table l). 8 



7 From the Chicago Challenge’s directories of grants and project records. 

8 We present school characteristics for the 1998-99 school year because the networks and schools 
funded at that time remained the Challenge’s core grantees through 2001. 
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Table 1. Characteristics of Chicago Annenberg Elementary Schools and Elementary 
Schools Citywide, 1998-1999 



ANNENBERG SCHOOLS CITYWIDE 

SCHOOLS 


Average student enrollment 


696 


706 


Low income 


85% 


85% 


English language learner 


18% 


18% 


Racial/ ethnic composition: 


African-American 


53% 


54% 


Latino 


33% 


34% 


White 


10% 


9% 


Asian/Pacific Islander 


3% 


4% 


Native American 


<1% 


<1% 


f 

1993 Eighth grade graduates who: 


Graduated from CPS high school 


40% 


40% 


Dropped out 


35% 


36% 


Left CPS 


25% 


24% 


Students in grades three through eight 
scoring at or above national 
norms on the ITBS: 


Reading 


36% 


35% 


Mathematics 


43% 


42% 



The Challenge supported relationships between schools and a large and diverse 
group of External Partners. Of the 45 External Partners working with Annenberg 
schools, about 35 percent were Chicago-area colleges and universities; 23 percent 
were arts and cultural institutions; and 28 percent were education reform and 
education services organizations. The remaining 14 percent of the Challenge's 
External Partners were neighborhood and community-based organizations. Almost 
two-thirds of all External Partners had some experience working with schools on 
long-term improvement projects, but one-third had no such experience. The names 
of Chicago Annenberg External Partners and the numbers of schools in their 
networks are listed in Appendix A. 

Annenberg grants supported a wide range of school improvement activity. About 
55 percent of the networks focused primarily on improving curriculum and 
instruction. Sixteen percent worked to improve student learning climate and social 
services for students and families. Another 13 percent were concerned primarily with 
developing parent and community support. The remaining 16 percent of Annenberg 
networks adopted more comprehensive foci to improve a number of areas 
concurrently, including curriculum and instruction, teacher professional community, 
school leadership, student learning climate, and parent and community support. 
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Within these general categories were a number of specific initiatives such as parent 
education programs, literacy programs, integration of arts and technology into the 
curriculum, health/science education, creating small schools, middle school 
restructuring, principal and teacher leadership development, and strengthening 
school-community ties. 

Figure 2 shows the total amount of financial support provided by the Challenge 
through implementation grants. 

Figure 2. Total amount of Annenberg Funds to Support School Improvement through 
Implementation Grants, 1996 to 2001 




As indicated in the figure, the total amount of funding grew considerably between 
1995 and 1999. This growth was associated not only with an increase in the number 
of schools that the Challenge supported, but also with an increase in the average 
amount of funding per school. In 1999, at its peak, the Challenge distributed $9.6 
million to support local school improvement activities. Between 1999 and 2000, 
however, the total amount of money distributed through implementation grants was 
reduced by almost 40 percent. By 2001, it was reduced further to less than one-tenth 
of the amount provided in 2000. These reductions in total funding occurred even as 
the Challenge continued to support more than 200 schools, albeit at rapidly 
diminishing levels. 

To get some sense of the implications of this decline in total funding, it is 
instructive to examine levels of average per school funding. Of course, different 
networks received different amounts of money and individual schools received 
varying amounts of money within and across their networks. Moreover, it is difficult 
without detailed analyses of network budgets to determine how much money was . 
used by External Partners to cover their own costs, to purchase goods and services for 



10 



The Chicago annenberc Challenge: Successes, Failures, and Lessons for the future 



schools, or to give directly to schools. Initial budget requirements set by the 
Challenge limited Partners to spend only 10 percent of any total grant to cover their 
own expenses. Later, however, the Challenge recognized that some Partners required 
more money to be effective and it altered this requirement to permit larger 
percentages of grants to be used by partners to cover their own expenses. 9 

Keeping this in mind, a rough measure of average annual per school funding was 
calculated based on the total amount of implementation grants awarded and the total 
number of schools within networks that received those grants. As shown in Figure 3, 
the average funding per school grew between 1996 and 1999 from about $15,000 to 
$47,000. Afterwards, this amount dropped considerably; from about $47,000 in 
1999, to $29,000 in 2000, to about $2,600 in 2001, leaving schools and Partners on 
average with almost no financial support. 

Even at its highest level, the average amount of per school funding made up only 
a small percentage of a typical elementary schoofs budget. In 1999, $47,000 
represented about 1.2 percent of the annual operating budgets of the elementary 
schools we studied in our field research. 10 This percentage does not take into account 
other grants these schools might have obtained, in which case the Annenberg 
proportion of the budget would have been even smaller. To look at it another way, 
the funds provided by the Challenge in 1999 — the peak year of network funding — to 
support an average size network of five schools amounted, in practical terms, to about 
enough to provide salaries, benefits, and support to two professional staff members. 

Figure 3 . Average Annenberg Funding Per School, 1996 to 2001 




9 Newmann and Sconzert (2000). 

10 Among Annenberg schools, the average annual budget was approximately $3,810,000; see 
Newmann and Sconzert (2000). 
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In addition to funding, the Challenge provided different forms of professional 
support to its schools and External Partners. In 1997, it sponsored workshops to help 
schools and Partners develop stronger school improvement plans and proposals for 
Challenge funding. Some schools and Partners received direct coaching on their 
program and proposal development. The Challenge also held workshops that year 
on the themes of time, size, and isolation, and on its vision of successful school 
improvement. External Partners from a few successful networks served as trainers 
and facilitators at these workshops. Also in 1997, in an effort to promote 
communication among schools and Partners, the Challenge printed the first of 
several directories listing its implementation networks and their member schools and 
External Partners. The directories also contained descriptions of the networks' 
primary activities. 

Later on, the Challenge sponsored another strand of workshops to provide 
networks with opportunities to share ideas and engage in joint problem solving. 
These workshops were also designed to bolster commitment to local improvement 
efforts. In addition, it sponsored presentations by outside speakers, some of whom 
were national figures in school reform. And finally, the Challenge organized fairs for 
schools and External Partners to display their work and celebrate their 
accomplishments. 

Providing these and other support activities was primarily the responsibility of 
one member of the Challenge staff, the Program Director. A Grants Manager and 
the Challenge’s Executive Director joined the Program Director in this effort. Both 
the Program Director and the Grants Manager had some, but not extensive, 
experience in school development. The Executive Director was hired from the local 
foundation community. His primary experience had been in grant making and 
community development. 11 



Breakthrough Schools 

In 1999, the Challenge identified 18 schools to receive sustained funding during its 
last two years to further promote their improvement and encourage them to serve as 
models and sources of support to other schools. The Challenge’s objective was to 
“[deepen] its work with schools that have demonstrated a readiness for reform.” The 



11 The entire Challenge staff consisted of an Executive Director who was hired in October 1995; an 
Office Administrator, who was hired in spring 1996; and a Program Director, Grants Manager, and 
Financial Officer, each of whom was hired in summer 1996. Between April and December 1997, the 
staff expanded to include a Director of Development, whose responsibility it was to help raise 
matching funds; a Communications Director and Assistant, who were to develop communication 
strategies and work with the local media; a Clerical Assistant, and a Data Manager. All told, relatively 
few staff resources were dedicated to provide professional support to schools and External Partners. 
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Challenge staff nominated schools for Breakthrough status relying on network 
progress reports, school visits, and records of school participation in Challenge 
activities. The Chicago School Reform Collaborative and the Donors Forum 
Education Group provided additional information. 12 Among their specific criteria 
was that schools be models of comprehensive, focused reform and be able to show 
evidence of gains in student achievement. Schools also had to be examples of strong 
teacher professional learning communities and strong school leadership. Moreover, 
they had to have had a strong record of participation in Challenge-sponsored 
activities. 

To become Breakthrough Schools, nominated schools had to make written 
requests to the Challenge. In their requests, schools had to indicate how they met the 
selection criteria and outline plans to deepen and extend their improvement efforts. 
Most schools proposed to use resources from the Challenge to deepen their 
commitment to teacher professional development, curriculum development, and 
student social support and learning climate. Breakthrough Schools were selected by 
Challenge staff in December 1999 and announced to the public in February 2000. 

As a group, Breakthrough Schools were similar to other Annenberg schools in 
size, student achievement, and demographic characteristics; however, they received 
substantially more funding during the Challenge’s last two years. While other 
Annenberg schools’ average funding dropped precipitously during this period to 
almost nothing, Breakthrough Schools were awarded nearly $100,000 or 

approximately $50,000 a year to support improvement activity (see Figure 4). It is 
important to note that Breakthrough funding went directly to the schools, not to the 
schools through their External Partners. As such, Breakthrough Schools had greater 
discretion over a somewhat larger sum of money than other Annenberg schools. 



12 Chicago Annenberg Challenge (1999). 
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Figure 4. Average Annenberg Funding per School, Breakthrough and All Annenberg Schools, 
1996 to 2001 




In addition to two more years of sustained funding, Breakthrough Schools 
received ongoing professional support from the Challenge. For example, in February 
2000, the Challenge organized a workshop for Breakthrough School principals on 
how to read and interpret individual school reports of teacher and student survey 
data prepared by the Consortium to assist with school improvement planning (these 
surveys are described later in this part of the report). Another workshop involved 
teaching faculty how to assess their classroom assignments in terms of the intellectual 
demands those assignments make on students. Other workshops aimed to help 
Breakthrough Schools write better grant proposals to support future improvement 
activities and communicate their accomplishments to the media and the larger 
community. Overall, by the end of the Challenge, the differences in funding and 
professional support provided to Breakthrough Schools stood in stark contrast to the 
funding and support provided to the other Annenberg schools. 



Relationship of the Challenge to the 
Chicago School System 

Recall that the Chicago Challenge was established to work “along side” of the 
Chicago public school system. Although it was designed to support local school 
improvement within the system and influence the direction of Chicago’s reform 
policy, it was never intended to be part of the system itself. Therefore, to understand 
the Challenge, it is important to understand Chicago’s reform agenda. Figure 5 
juxtaposes the Challenge’s development with that of key school reform initiatives 
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developed by CPS and the Illinois General Assembly. 13 This figure is not meant to be 
all-encompassing, only to depict key events that describe each. 

The Chicago Challenge was designed according to many of the principles of 
democratic localism and grassroots action that defined Chicago’s 1988 
decentralization reform (Illinois PA85-1418 School Reform Act). It sought to extend 
the work of what is considered Phase I of Chicago school reform from governance to 
other areas of school improvement. It is important to note that when the Challenge 
was designed, it was assumed that the then current central administration would be 
in place for the foreseeable future and that decentralization and local school 
governance would be the foundation for school reform for some time to come. 

Six months after the Challenge was established, everything changed. The Illinois 
legislature ushered in Phase II of reform when it passed an amendment to the 1988 
school reform bill, the Illinois HB206 School Reform Act. This amendment 
restructured the CPS central administration around a corporate-style management 
team that included a Chief Executive Officer in place of the superintendent and a 
five-member Reform Board of Trustees appointed by the Mayor. The amendment 
established greater accountability within the system by clarifying and extending the 
authority of the CEO to intervene in nonimproving schools. 



13 For more detail on school reform in Chicago, see Bryk et al. (1998a); Hess (1991, 1993); and 
Shipps, Kahne, and Smylie (1999). For a detailed description of the influence of Chicago school 
reform on the development of the Chicago Annenberg Challenge, see Shipps and Sconzert with 
Swyers (1999). 
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Figure 5. Key Events in the Chicago Annenberg Challenge and in Chicago School Reform Policy, 
1988 to 2001 
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As the Chicago Challenge began awarding its first implementation grants, the 
new central administration introduced two major initiatives to bring centralized, 
high-stakes accountability into the system. It placed schools with fewer than 15 
percent of students scoring at or above national norms on the Iowa Tests of Basic 
Skills (ITBS) on academic probation and assigned each a probation partner and a 
probation manager to direct school improvement efforts. Schools on probation that 
failed to improve their test scores over a period of time could be reconstituted. The 
administration also developed a new policy to end social promotion. Students in the 
third, sixth, and eighth grades were required to meet specified cut-off scores on the 
ITBS in order to advance to the next grade level. If they failed to meet these 
benchmarks, they had to attend mandatory summer school and, if they failed again 
to achieve the cutoff scores at the end of the summer, they were retained at grade 
level. 

A year later, the administration developed newsystemwide goals and standards 
for student achievement. It began to create lesson plans keyed to these standards and 
curriculum-specific examinations for high school graduation. A major capital 
improvement initiative was begun to build new schools, repair and renovate existing 
facilities, and alleviate overcrowding. CPS established the Lighthouse program to 
provide after-school academic, recreational, and social learning opportunities for 
students, and began to place new emphasis on early childhood education. 
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Against the backdrop of centralized initiatives and high-stakes test-driven 
accountability, the Challenge was encouraging its schools and External Partners to 
address the organizational issues of time, size, and isolation. Later, as more and more 
attention was placed on student academic achievement and test score performance, 
the Challenge encouraged its grantees to intensify their focus on teaching, learning, 
and whole school change in general and intellectually challenging instruction and 
teacher professional development in particular. Developed to build upon and extend 
decentralized school reform in Chicago, the Challenge began to experience conflict 
with the school system’s reform initiatives emphasizing uniform performance 
standards and centrally imposed sanctions. 1 ^ The ground had shifted, and the 
Challenge found itself eclipsed by a highly visible central administration, the Mayor’s 
office, and a business community and local media that were largely supportive of the 
new administration s initiatives. According to the Challenge’s Executive Director, the 
Challenge was “not the elephant in the town .” 15 

There were, of course, areas where the school system’s initiatives and the 
Challenge’s efforts were compatible and mutually reinforcing. An earlier Chicago 
Annenberg Research Project report provided some examples where they supported 
each other . 16 For instance, the system’s capital improvement efforts were 
instrumental in improving learning climates in several of the schools we studied. 

Nevertheless, the Challenge promoted a reform agenda that often collided with 
specific system policies, which created tensions and dilemmas for principals and 
teachers at the school and classroom levels. Nowhere was this more sharply 
pronounced than in the interaction between high-stakes standardized testing and 
efforts to improve instruction. Early field research documented examples of schools 
where high-stakes testing, coupled with the system’s probation and student retention 
policies, played a crucial role in catalyzing a press for accountability and a perceived 
need for change . 17 These policies moved some schools from complacency into action. 
At the same time, the field research identified examples of other schools in which 
high-stakes testing pushed teachers and principals in low-achieving schools to focus 
on the quickest means of administrative compliance that was at hand — test 
preparation — and to abandon or push aside at least for a while efforts to achieve 
more ambitious, long-term instructional improvement. 

Structurally and politically, the Challenge had difficulty developing a close and 
productive working relationship with the CPS central administration. The 



14 Shipps and Sconzert with Smylie (1999). 

15 ibid. 

16 Wenzel et al. (2001). 

17 ibid. 
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relationship was tenuous at best; for the most part it was strained and at times it was 
antagonistic. Top system administrators did not fully trust the Challenge's 
leadership, whom they associated with the “failed" efforts of decentralization reform. 
These administrators were uncomfortable with their inability to control the largest 
independent reform initiative operating within the system and its substantial 
resources. And, although the Challenge's leadership sought to cultivate a working 
relationship with the CPS central administration, it also made no secret of its intent 
to influence the system, sometimes using the local media to expose flaws it perceived 
in CPS policies and practices. Indeed, a number of persons associated with the 
development and operation of the Challenge were openly critical of the system's 
leadership and its initiatives. 

In spring 2001, the system's Chief Executive Officer, who was appointed in 
1995, resigned and a new central administration was appointed. The new 
administration focused more attention and resources on instructional improvement, 
creating new initiatives in reading, teacher professional development, and leadership 
development. These initiatives signaled a new direction, a Phase III of school reform 
in Chicago. Ironically, just as the reform agendas of the system and the Challenge 
began to converge, the Challenge reached the end of its operation. Several 
implications of the Challenge's relationship with CPS and its reform agenda are 
explored at the end of this report. 



How the Study Was Conducted 

The research on which this report is based was organized around an elaborated 
conceptual framework of school development and a multi-method research design. 
This framework, the Model of Essential Supports for Student Learning, identifies 
areas of school organization and practice that have been shown both in the literature 
and in other research performed by the Consortium on Chicago School Research to 
promote student learning. 

The research design was composed of four related strands of inquiry: (a) 
longitudinal field research in a sample of Annenberg elementary schools; (b) 
documentation of the Chicago Annenberg Challenge as a reform initiative and as an 
organization; (c) analyses of systemwide teacher, student, and principal survey data; 
and (d) analyses of standardized test scores. Field research was used to document 
improvement in specific areas of school organization and practice and to gather 
evidence of how improvement was achieved. It was also used to document the 
Challenge's support of local school improvement. To this micro-level work was 
added a scaffold of survey research and analyses of student standardized test scores. 
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These macro-level analyses were conducted to identify patterns of improvement in 
the Essential Supports and student outcomes across Annenberg schools as a whole. 
They were also used to compare patterns of improvement and student outcomes in 
Annenberg schools to patterns across demographically similar non-Annenberg 
schools. Detailed information about the research methodology can be found in the 
appendices. 

Both field research and survey data analyses were used to answer the first central 
question addressed by this report — Did the Chicago Annenberg Challenge promote 
improvement of the schools it supported? Analyses of survey data and student test 
scores were used to answer the second central question — Did the Challenge promote 
improvement in student achievement and nonacademic outcomes in those schools? 
Field research and descriptive survey data were used to address the third 
question — What factors might explain improvement or the lack thereof among 
Annenberg schools? Finally, findings from all strands of inquiry were used to address 
the fourth question — What can we learn from the Challenge's experience about 
promoting school improvement? 



Model of School Development 

School improvement can mean many different things. Unlike the more general 
concept of change, to say that a school has improved implies that it has changed in 
some positive, valued direction. Ffowever, there are any number of positive, valued 
directions for school change that might be considered improvement. Not articulating 
what those directions are may render the study of school improvement ambiguous 
and without much meaning. Therefore, it was important to define school 
improvement at the beginning of the research to determine how improvement by 
that definition may have occurred. 

The definition of school improvement used in the Chicago Annenberg Research 
Project proceeded from the goal of increasing student academic learning. The type of 
student academic learning with which the project was primarily concerned included 
the acquisition of basic knowledge and skills, but went further to include deeper 
understanding of subject matter and students’ ability to produce “authentic” 
intellectual work . 18 This involves the development of cognitive capacities that allow 
students to work with existing knowledge and to create new knowledge to analyze 
and solve real-world problems, manage personal affairs, and become economically 
productive and responsible members of society. Following from the goal of increasing 
this type of student academic learning, school improvement was defined in terms of 



18 Newmann, Bryk, and Lopez (1998). See also National Research Council (2000) and Newmann and 
Associates (1996). 
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those aspects of school organization and practice that, when strengthened, would 
most likely promote such learning among students. 

The Model of Essential Supports identifies seven areas of school organization and 
practice that support such intellectually ambitious academic learning: (a) high-quality 
instruction; (b) supportive student learning climate; (c) school leadership; (d) teacher 
professional community; (e) parent and community involvement; (f) relational trust; 
and (g) instructional program coherence (see Figure 6). 



Figure 6. Model of Essential Supports for Student Learning 




instructional 
Program Coherence 



Relational trust 



According to the model, schools are said to have improved when they have 
developed from lower to higher (or weaker to stronger) states on one or more of the 
Essential Supports. When schools are said to have failed to improve, they have made 
no progress from lower to higher states on the supports. Finally, when schools are 
said to have worsened or regressed, they have fallen from higher to lower states of 
development. Each of the Essential Supports and their states of development are 
described in more detail in Part Two of this report. Specific indicators of high and 
low states of development of each are presented in Appendix B. 

The model specifies that each support, when developed, may serve to promote 
student academic learning. Implicit in the model is a logical “ordering” of the 
supports in their relationship to one another and to student learning. This ordering, 
which reflects the literature on academically effective schools, suggests that student 
academic achievement is most likely to be promoted by developing those supports 
most proximal to students’ learning — high-quality instruction and supportive 
student learning climate. 19 Although the other supports in the model may contribute 
in some direct ways to improving student academic learning, their influence is more 
likely to be indirect, providing the organizational conditions necessary to develop and 



59 See Good and Brophy (1997) and Wang, Haertel, and Walberg (1993). 
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support instruction and learning climate. For example, even though school leadership 
and teacher professional community may both play an important role in improving 
student academic learning, that role may be more indirect than direct through their 
respective and related influences on the development of instruction and learning 
climate. Likewise, the two “overarching” supports in the model — relational trust and 
instructional program coherence — may also influence student academic learning 
indirectly, providing the social and structural bonding to hold together the other 
organizational and practice supports and direct them toward improving student 
learning. 

Such a logical ordering suggests that without developing quality instruction and a 
supportive student learning climate, it is unlikely that a school would be able to 
achieve substantial, sustained improvement in student academic learning. It also 
suggests that it would be unlikely that a school could achieve much in the way of 
developing quality instruction and student learning climate without antecedent or 
concurrent development of the elements of school organization required to facilitate 
high-quality instruction and a supportive student learning climate. This suggests the 
possibility that a school may show signs of improvement in developing the 
organizational supports of leadership, professional community, parent and 
community involvement, relational trust, and program coherence that are arguable 
antecedents to quality instruction and supportive student learning climates but not 
yet show signs of improvement in the latter two supports. Likewise, a school may 
show signs of initial development of the Essential Supports, including quality 
instruction and student learning climate, without those supports having developed 
sufficiently and for a long enough time to result in improved student achievement. 

The Model of Essential Supports was selected for this research for several reasons. 
First, it has strong support in the empirical literature on academically effective 
schools and school improvement and is being validated by ongoing analyses at the 
Consortium on Chicago School Research . 20 Second, the model is well established in 
Chicago public schools. It had served as a template for local school improvement 
planning for several years prior to the Challenge and had been adopted by CPS as a 
model for principal leadership development. Third, and most importantly for this 
study, the model was consistent with and inclusive of the wide range of local school 
improvement goals and activities supported by the Chicago Annenberg Challenge . 21 



20 See Bryk et al. (forthcoming); Designs for Change (1993); Newmann et al. (1998); Newmann, 
Bryk, and Nagaoka (2001); and Wenzel et al. (2001). 

21 Chicago Annenberg Challenge (2000) and Hallett et al. (1995). 
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Longitudinal School-Level Field Research 

At the center of this study was longitudinal school-level field research conducted 
from the 1996-97 school year through 2000-01. The initial field research sample 
consisted of 23 elementary, middle, and high schools in 10 Annenberg networks. 

Due to the importance of networks in the Challenge’s initial strategy, field research 
schools were selected on the basis of their participation in certain networks that were 
chosen because they represented different school improvement emphases (e.g., 
curriculum and instruction and parent and community involvement) and had 
different types of External Partners (e.g., universities, community organizations, and 
cultural institutions). Consideration was also given to select some networks that were 
newly formed and networks that were built on well-established relationships between 
schools and the External Partners. 

Once the 10 networks were chosen, two or three schools from each were 
identified to serve as research sites. One to two of these schools were identified 
because of their promise for working well with their External Partners and succeeding 
in their efforts to develop. An additional school was chosen because of indications 
that it might struggle to succeed. The intention was to create a purposive sample of 
schools that would provide points of comparison and contrast to understand reasons 
for more and less successful improvement. School selections were informed by 
previously collected Consortium survey data and by assessments from the External 
Partners of the networks that were sampled. 

Field research schools were sampled in two stages. The first group of schools was 
selected in fall 1996 from the networks that received implementation grants during 
the Challenge’s first round of grant making. The second group was selected in fall 
1997 from networks that received implementation grants in the second round. 

Field data were collected from all 23 schools in this initial sample in the 1996-97 
or 1997-98 school years, depending on when the school entered the Challenge, 
and then again in the 1998-99 school year. After that year, the sample was reduced 
to 14 elementary schools. Among the nine schools dropped from the study, several 
chose not to continue in the project. For others, there was such little school 
improvement activity that subsequent data collection activity would have yielded very 
little useful information. From 1999-00 through 2000-01, data collection proceeded 
in these 14 schools. During these two years, two of these 14 schools failed to provide 
adequate data for cross-school comparisons. Due to lack of data from these two 
schools, we focused our qualitative analysis of school development on the 12 
elementary schools from which full longitudinal data were obtained from the 1 996- 
97 or 1997-98 school year through 2000-01. 



22 



The Chicago annenberg Challenge: Successes, Failures, and Lessons for the Future 



The final research sample of 12 elementary schools was quite similar in 
characteristics to the initial full sample. Half were in networks from each round of 
initial funding (either in 1 996 or 1997). Like the initial field research sample of 23 
schools, the schools in the final sample were generally typical of schools across the 
Challenge and the system as a whole, although their average student enrollment was 
somewhat larger (see Tables 1 and 2). 

Several types of data were collected from each field research school by a lead 
researcher (typically a university faculty member or advanced doctoral student) and a 
research assistant. These data were collected during either the 1996-97 or 1997-98 
school years (depending on when the school’s network first received funding) and in 
both 1998—99 and 2000—01. Data included (a) classroom observations of six 
language arts teachers and six mathematics teachers, two each from the third, sixth, 
and eighth grades; (b) classroom observations of two or three additional teachers 
involved with specific Annenberg initiatives; (c) samples of instructional assignments 
and student work in reading/writing and mathematics from the observed classrooms; 
(d) interviews with each observed teacher, the principal, the school’s External Partner 
and coordinator, the LSC chair, an LSC teacher representative, a member of the 
school’s Professional Personnel Advisory Committee, and the teacher union 
representative; (e) observations of meetings and events associated with the school’s 
Annenberg activities and other major school improvement initiatives; and (f) 
documents pertaining to school improvement and to Annenberg network 
membership and activity. 
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Table 2. Characteristics of First Sample of Twelve Field Research Schools, 
1998-1999 



r — — 


FIELD RESEARCH SCHOOLS 


Average student enrollment 


888 


Low-income 


89% 


English language learners 


21% 


Racial/ ethn ic compositio n : 


African-American 


50% 


Latino 


41% 


White 


7% 


Asian/Pacific Islander 


<1% 


Native American 


0% 




FIELD RESEARCH SCHOOLS 


1993 8th grade graduates who: 


Graduated from a CPS high school 


39% 


Dropped out 


39% 


Left CPS 


22% 


Percent of students in grades 3 through 8 scoring at or above national norms 
on the I TBS: 


Reading 


32% 


Mathematics 


37% 



Field researchers were responsible for documenting the development of each 
Essential Support in their schools, as well as the activities in which schools engaged 
to get better. Interviews were audio taped and transcribed. Observation notes, 
documents, and other materials were organized and archived. Researchers wrote 
structured descriptive case reports of their schools 5 development at three points in the 
project — 1996-97 or 1997-98, 1998-99, and 2000-01 — and wrote vignettes that 
described schools 5 efforts to get better. Cases and vignettes of all field research 
schools were read and coded independently by three research analysts. 

Discrepancies in coding were discussed and reconciled by the analysts through 
consensus procedures. Using this process, field research schools were classified as 
“developing 55 or “nondeveloping 55 and specific areas of development were 
categorized. These designations and the themes and patterns of school development 
that were identified across the sample were presented back to field researchers for 
validation. Analysts identified specific examples of school development and activities 
to promote that development to illustrate themes and patterns in both the broader 
field research data and findings from survey data. These examples were also shared 
with and confirmed by field researchers. See Appendix C for additional information 
about the field research methodology. 



O 

ERIC 






40 



24 



The Chicago Annenberc Challenge: Successes, Failures, and lessons for the future 



Documentation of the Chicago Annenberg Challenge 

Study of the Chicago Annenberg Challenge as both a large-scale reform and an 
organization relied heavily on documents produced by the Challenge itself including 
those associated with its founding, its requests for proposals, meeting notes, records 
of grant making, and correspondence. In addition, between June 1996 and 1998, a 
member of the research project staff observed nearly all the Challenge's formal 
meetings and events and then observed samples of meetings and events through 
2001. School-level descriptive data and Challenge grant records were used to identify 
patterns of decision making. 

In addition, research project staff formally interviewed and spoke regularly with 
the Challenge's Executive Director, Program Director, and Grants Manager. 
Members of the Challenge's Board of Directors and members of the Chicago 
School Reform Collaborative, a group that helped organize and manage the 
Challenge in its first year, were also interviewed. In 1997, 70 organizational leaders 
from seven sectors in the Challenge's institutional environment were interviewed 
about the Chicago Annenberg initiative and school reform in Chicago. These sectors 
included business, community, foundation, government, higher education, labor, 
and media. Finally, 19, or about 45 percent of External Partners were interviewed in 
1997, 1999, and 2001. Together, these interviews provided perspectives on the 
Challenge as a reform and an organization from both “inside” and “outside” the 
Challenge. 



Surveys of Teachers, Students, and Principals 

This study used survey data from teachers and students across the system to map the 
development of the Essential Supports among Annenberg schools and to compare 
that development to development found in demographically similar non-Annenberg 
schools. Student surveys were used to assess student social and psychological 
outcomes. The Consortium administered these surveys in the spring of 1994, 1997, 
1999, and 2001 (survey samples are described in Appendix D). Surveys from 1994 
and 1997 established baseline data and the 1999 and 2001 surveys provided data to 
track changes. Rasch measures were developed from individual survey items as 
indicators of various elements of the Essential Supports. Appendix E contains full 
descriptions of these measures. 

Hierarchical linear models were used to track changes in the Essential Supports 
and student outcome measures over time from baseline years, and to assess 
differences between (a) Annenberg and non-Annenberg schools and (b) 
Breakthrough Schools and other Annenberg schools. These analyses controlled for a 
number of school characteristics including school racial and ethnic composition, 
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school level of achievement, school size, and percent of low-income students (see 
Appendix D). Tests were also made of network effects on school development. It 
seemed reasonable to assume that schools in networks that focused primarily on one 
area of school development might be more likely to show changes in that area than 
schools in networks that focused on other areas. In 1999, the year with the most 
overall change in Annenberg schools, we tested for network-level differences by 
primary network focus. No statistically significant differences among network foci 
were found, suggesting perhaps greater within-network than between-network 
variation in development. Therefore, the study’s focus shifted from looking for 
network-level effects to examine school development across all Annenberg schools. 

There are numerous complexities in trying to create a single indicator of school 
development or making general statements about the overall development of a school 
or a group of schools. A school may develop on one or more of the Essential 
Supports but not on others. Moreover, a school may develop on some aspects of a 
particular support but not others. For example, a school may have a strong and active 
parent group, but its principal may lack the ability to involve it effectively in the life 
of the school. A school may increase professional development opportunities for 
teachers but at the same time experience erosion in the overall quality of the 
professional development and a decline in teacher participation. A school may make 
great strides in developing a strong, caring, personal student learning climate, but 
make little progress in raising expectations for student achievement or improving the 
quality of classroom instruction. A school may have an excellent relationship with its 
External Partner, but frustrate the Partner’s work by adopting contradictory and 
competing improvement initiatives. And so on. 

In order to deal with such possibilities, we examined change in each of the 
Essential Supports and change in different aspects of the same support separately. For 
instance, rather than considering change in teacher professional community as a 
single construct, change was examined with respect to each dimension of professional 
community (e.g., teacher collaboration, collective responsibility for student success, 
teacher innovation, and teacher commitment). The assumption was that if most or 
all aspects of professional community were found to have changed in a similar 
direction with statistical significance, some general conclusion about overall 
development of professional community could be drawn. 

The survey data provided two baseline points — spring 1994 and spring 
1997 — from which to assess development of Annenberg schools through spring 
2001. Spring 1994 data formed a baseline point prior to the establishment of the 
Challenge. Spring 1997 data formed a baseline point aligned with the first full school 
year of implementation grant funding. For the vast majority of measures for which 
there are 1997 data, there are also 1994 data. In order to show long-term change 
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across Annenberg schools, 1994 was chosen as the primary baseline point for analysis. 
For the few measures for which 1994 data do not exist, 1997 was used as the baseline 
point. Analyses examined overall patterns of change between 1994 and 2001 and 
intermediate patterns of change between 1997 and 1999 and between 1999 and 
2001. It is important to note that in 1994, there were no statistically significant 
differences between Annenberg and demographically similar non-Annenberg schools 
on any measure of the Essential Supports. Unless otherwise noted, differences are 
considered statistically significant if they occur at the 0.01 level ( p < 0.01); that is, if 
there is less than a 1 percent likelihood of them occurring by chance. 22 

Systemwide principal survey data were used to examine Annenberg principals' 
experiences with and perceptions of the Challenge as an organization, their schools' 
External Partners, and the support each provided. Data from principal surveys were 
also used to describe the role that the Challenge played in Annenberg schools and the 
level of their schools' participation in Annenberg activities. Principal surveys were 
administered in spring 1997, 1999, and 2001. 



Analyses ofITBS Scores 

ITBS scores were used as the primary indicator of student academic achievement and 
were analyzed in several different ways. Yearly rates of gain in Grade Level 
Equivalents (GEs) were used to map trends in reading and mathematics achievement 
at and across grade levels in Annenberg schools. Annual GE gains from 1994 through 
2001 were calculated in both reading and mathematics for students in Annenberg 
schools in grades three through eight. GE gains from the same period were also 
calculated for the same grades in demographically similar non-Annenberg schools. In 
this way, gains in Annenberg schools could be compared to gains in schools that did 
not participate in the Challenge. In order to assess achievement trends in 
Breakthrough Schools, GE gains in reading and mathematics were calculated and 
compared for students in grades three through eight in both Breakthrough and other 
Annenberg schools. Rather than looking at simple trends in average test scores, 
academic achievement was assessed using rates of gain. This was done on the 
assumption that if the Challenge was successful in improving academic achievement, 
one might expect to see an accelerating and growing difference between Annenberg 
and non-Annenberg schools in the size of gains over time. 



22 Given the number of statistical tests that were performed, this p-value was used to compensate for 
the possibility of Type 2 errors. A 0.01 p-value is more conservative than a 0.05 value, but still liberal 
enough to not miss important differences. 
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GE gains are a familiar and useful indicator to identify trends in academic 
achievement, but a more rigorous indicator was used to test the statistical 
significance of differences in achievement between Annenberg and non-Annenberg 
schools and between Breakthrough and non-Breakthrough Annenberg schools. 

These comparisons used an index of academic productivity developed by the 
Consortium. 23 This index measures the extent to which schools extend, sustain, or 
fail to sustain student learning achieved at previous grade levels over time. This index 
is built using gains in the ITBS scores of students who are enrolled in a school for at 
least one full academic year and helps account for the effects of student mobility on 
school-level achievement. The index takes into account students' past academic 
achievement, as measured by their ITBS score the previous year, and it takes into 
account effects of different ITBS test forms. The index measures achievement gains 
in both reading and mathematics in grades three through eight from 1992 through 
2001. 

Regression analyses were used to compare different groups of schools on the 
productivity index. These analyses used school group membership — Annenberg 
versus non-Annenberg and Breakthrough versus other Annenberg — as the key 
independent variable and the productivity index as the dependent variable. These 
analyses controlled for the size, neighborhood socioeconomic level, and racial and 
ethnic composition of the school, among other variables. They determined whether 
student achievement differed depending on whether a school was or was not an 
Annenberg school, or whether it was or was not a Breakthrough school. Unless 
otherwise noted, if differences between Annenberg and non-Annenberg schools or 
between Breakthrough and other Annenberg schools had occurred at the 0.01 level (p 
< 0.01); that is, if there was less than a 1 percent likelihood of them occurring by 
chance, it was concluded that there were statistically significant differences between 
the groups of schools being compared. 24 More information about the productivity 
index and these analyses is contained in Appendix F and Appendix G. 



Considerations 

There are at least three issues concerning aspects of this research methodology that 
should be considered. The first concerns self-report data collected through survey 
questionnaires and interviews. Such data may be subject to two types of problems 
that challenge validity — the difficulty that respondents may have representing a 
particular phenomenon accurately, and the possibility that because of self-interest, 
respondents may be positively or negatively biased in their perceptions and reports. 



23 For more information about the development of the productivity index, see Bryk et al. (1998b). 

24 As discussed in the next section, differences between Breakthrough and other Annenberg schools 
that occur at the 0.05 level are reported as part of a broader pattern of findings. 
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Three strategies were used to reduce the potential for these problems. 25 A pattern- 
matching strategy was used whereby findings, particularly those from survey data, 
were examined to determine whether they were consistent with what is already 
known about schools and school change from existing theoretical and empirical 
literature. In addition, data from different sources about the same phenomena were 
triangulated” for consistency. For example, data from surveys were compared to data 
from field research and documents and interview data were compared to 
documentary and observational data. Finally, findings were presented to and verified 
by the field researchers, those persons most familiar with the schools being studied 
and in the best position to identify biases and inaccuracies in self-reported data. 

A second issue was that of disentangling the influence of the Challenge on school 
change from other sources of potential influence. As noted, Annenberg schools were 
like many other CPS schools in that they were involved in multiple improvement 
projects. Some worked simultaneously with several external organizations in addition 
to their Annenberg External Partners. Some of the sources of greatest potential 
influence on school change were CPS policies, particularly the high-stakes testing and 
probation policies. In 1999, 54 of Annenberg’s 206 elementary schools, or 26 
percent, were on academic probation because of low standardized test scores. 
Systemwide that year, 91 elementary schools, or 16 percent of all elementary schools 
were on probation. Schools on academic probation were required to have a probation 
partner to help them improve. Of the 54 Annenberg schools on probation, about 20 
worked with Annenberg External Partners who also served as their probation 
partners. The remaining 34 Annenberg schools on probation had different 
Annenberg and probation partners. These schools represented only 16 percent of all 
Annenberg schools and, because this proportion was relatively small, the issue of 
entanglement of Annenberg and probation partner influences is probably not very 
significant. Moreover, because of the large number of schools receiving Annenberg 
support and because of their similarity to schools across the system, non-Annenberg 
sources of influence are likely to have been distributed similarly across Annenberg 
and non-Annenberg schools. 

Still, there are several aspects of the research design and methodology that help to 
strengthen conclusions about an “Annenberg effect.” Using school achievement level 
as one of many statistical controls helped account for the effects of probation on both 
Annenberg and non-Annenberg schools. In addition, the field research revealed 
much about the influence of the Challenge and its External Partners compared to 
other sources of influence, including CPS policies. Such distinctions are documented 
and discussed in several places in this report. 



25 See Merriam (1998); Stake (1995); and Yin (1989). 
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A third issue concerns that of the significance of the research findings. Some of 
the statistically significant changes and differences between groups of schools are 
quite small. On the one hand, because the statistically significant differences that are 
described in this report are based on averages of hundreds of schools and thousands 
of teachers and students, even the smallest non-chance differences should be 
considered real and meaningful. On the other hand, it can be argued that even 
though they may be statistically significant, small non-chance differences may not be 
very meaningful or educationally significant. There is ongoing debate about this 
matter in the literature . 26 Nonetheless, it is important to consider whether small, 
statistically significant differences across a very large number of schools are 
educationally significant because of the difficulty and length of time it takes to 
change so many schools, or whether these differences are on average so small that for 
all practical purposes they mean very little in the daily experiences of individual 
schools. 



26 See Berliner (1987). 
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Our research findings are presented in four sections. The first two concern 
the Challenge’s “bottom line” improvement in student academic 
achievement and non-academic student outcomes — and how Chicago 
Annenberg schools developed in ways that might promote student learning. 
Both show how changes among Annenberg schools compare to changes 
among demographically similar schools that did not participate in the 
Challenge. The third section presents findings on student outcomes and 
school development in the Breakthrough Schools. We conclude with an in- 
depth look at improving and non-improving Annenberg schools to 
understand the factors that make individual local school improvement 
successful. Details of the statistical findings presented here are contained in 
Appendices G and H. 



Student Outcomes 

As described in Part One, our primary measure of student academic achievement was 
rates of gain on the reading and math portions of the ITBS. In addition, four social 
and psychological student outcomes were examined: (a) academic engagement in 
school, (b) sense of self-efficacy, (c) classroom behavior, and (d) social competence. 
Academic engagement refers to students’ interest and participation in learning and 
whether they work hard to do their best in school. Sense of self-efficacy refers to 
students’ confidence in their own academic abilities and their perceptions of their 
chance for success on even the most difficult work. Classroom behavior is the extent 
to which students in a classroom respect each other, work well together, and help 
each other learn in addition to the degree of student disruption of classroom activity. 
Finally, social competence refers to students’ sense that they listen well to what others 
have to say; share, help, and work well with each other; and help resolve arguments. 
Data from the Consortium’s 1994, 1997, 1999, and 2001 student surveys were used 

30 




47 



Part Two: Findings 31 



to examine changes in these outcomes and to test for differences between Annenberg 
and demographically similar non-Annenberg schools. 



Achievement on the ITBS 

Analyses of ITBS scores reveal that overall, student achievement in Annenberg 
schools rose between 1996 and 2001 (see Appendix G). During this period, reading 
achievement rose an average of 1.01 GEs across grades three through eight. Math 
achievement rose an average of 0.95 GEs. These increases are consistent with those 
reported for the system as a whole. 27 

Although student achievement increased in Annenberg schools, the rate or size of 
gains did not markedly improve. Across grade levels, the size of one-year gains in GEs 
remained constant or fluctuated only slightly. This pattern held true for both reading 
and math achievement, though overall gains in reading were slightly larger than gains 
in math. Some differences in the size of gains were found at different grade levels. In 
reading, average GE gains were lower in the third and sixth grades than in other 
grade levels; in math, average GE gains were lower in the third and seventh grades. 
These are consistent with the rates of gain across the system as a whole. 28 

Findings from the third and sixth grades illustrate trends in student achievement 
gains in Annenberg schools. GE gains in third-grade reading held relatively constant 
between 1996 and 2001 (see Figure 7). These slight fluctuations are not noteworthy 
considering the different ITBS test forms used during this period. This pattern is also 
present in third-grade math and sixth-grade reading, though the size of sixth-grade GE 
gains were generally larger than third-grade gains (see Figures 8 and 9). Sixth-grade 
math gains followed the same pattern of no net gain but slight fluctuation (see Figure 
10). 

Analyses using the productivity index reveal that achievement trends in 
Annenberg schools did not differ from those in demographically similar non- 
Annenberg schools. There were no statistically significant differences in reading or 
math at any grade level in any year between 1995 and 2001. Although Annenberg 
schools appeared to outperform non-Annenberg schools in some years at particular 
grade levels, the reverse appeared to occur in other years. None of these differences 
were statistically significant. 



27 Rosen k ran z (2002). 

28 ibid. 
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Figure 7. Grade Equivalent Gains on the ITBS in Annenberg and Demographically Similar Non- 
Annenberg Schools: Third-Grade Reading, 1994 to 2001 
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Figure 8. Grade Equivalent Gains on the ITBS in Annenberg and Demographically Similar Non- 
Annenberg Schools: Third-Grade Math, 1994 to 2001 




Figure 9. Grade Equivalent Gains on the ITBS in Annenberg and Demographically Similar Non- 
Annenberg Schools: Sixth-Grade Reading, 1994 to 2001 
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Figure 10. Grade Equivalent Gains on the ITBS in Annenberg and Demographically Similar Non- 
Annenberg Schools: Sixth-Grade Math, 1994 to 2001 
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Social and Psychological Outcomes 

Trends in different student social and psychological outcomes in Annenberg schools 
were mixed. Student academic engagement improved while students' sense of self- 
efficacy, classroom behavior, and social competence weakened. Like changes in 
academic achievement, changes in social and psychological outcomes among 
Annenberg schools were similar to changes in demographically similar non- 
Annenberg schools; there were no statistically significant differences on any outcome. 

Table 3 groups student outcome measures according to whether they improved 
or weakened between the baseline year of 1994 or 1997 (depending upon when data 
were first available) and 2001 as defined by substantive categories of the measures 
(e.g., limited, moderate; see Appendix D). Changes are shown in terms of (a) 
measure categories; (b) differences in means on the 10-point Rasch scale used to 
construct each measure; and (c) standardized change unit differences, which show 
differences in terms of standard deviations in the baseline year (see Appendix H). 
Finally, the table shows how Annenberg schools, on average, compare to 
demographically similar non-Annenberg schools on each outcome. Line graphs based 
on standardized change unit differences from the baseline year also illustrate the trends. 
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Table 3 . Student Social and Psychological Outcomes in Chicago Annenberg Schools, 1994 or 
1997 to 2001: Summary of Findings 





1994 OR 1997 


2001 


DIFFERENCES 
IN MEANS 


STANDARDIZED 
CHANGE UNIT 
DIFFERENCE 


NON- 

ANNENBERG 

COMPARISON 


Improved 


Student 

Academic 

Engagement 


Limited 


Moderate 


+ 0.08 


+ 0.31 




Weakened 


Student 
Sense of Self- 
Efficacy 


High 


High 


- 0.08 


-0.30 




Student 

Classroom 

Behavior 


Moderately 

positive 


Moderately 

positive 


-0.11 


- 0.50 


gg 


Student 

Social 

Competence 


Moderate 


Moderate 


-0.22 


- 1.05 


gg 



Note: Measures are considered improved or weakened if the difference in means between 1994 or 1997 and 2001 is statistically 
significant at or beyond 0.01 {p < 0.01). Comparisons to non-Annenberg schools are for 2001. A “+” indicates that Annenberg 
schools were stronger than non-Annenberg schools on a particular measure {p < 0.01). A “e” indicates that the two groups of 
schools were statistically equivalent. A indicates that Annenberg schools were weaker than non-Annenberg schools on the 
measure [p < 0.01). 



Student Academic Engagement. Engagement in Annenberg schools rose 
between 1994 and 1997, then declined slightly between 1997 and 1999, and 
remained steady between 1999 and 2001 (see Figure 1 1). Overall, student academic 
engagement in Annenberg schools was greater in 2001 than in 1994, although the 
difference was quite small. Still, the net difference did move the average level in 
Annenberg schools from the very high end of the measure’s “limited” category to the 
very low end of “moderate” category. Students were somewhat more likely in 2001 
than in 1 994 to report that they worked hard to do their best, that the topics they 
studied were interesting, that they were not often bored in class, and that they were 
interested in what was going on in class. For several years, student academic 
engagement in Annenberg schools was slightly greater than engagement in 
demographically similar non-Annenberg schools. None of these differences were 
statistically significant, however. 

Student Sense of Self-Efficacy. In 1997, students’ sense of self-efficacy in 
Annenberg schools was “high” but not “very high.” That year, students were likely to 
report that they cared if they got bad grades in school, felt they could do better, and 
believed they could do a good job if they had enough time. They were also likely to 
report that they could complete the hardest work they were assigned if they tried and 
that they were certain they could master the skills taught in class. They were mixed in 
whether they thought they could understand all class work even if they tried hard. 
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Levels on this measure in Annenberg schools fell between 1997 and 1999, but rose 
slightly between 1999 and 2001 (see Figure 12). Despite this improvement, they 
remained lower in 2001 than in 1997. There were no statistically significant 
differences in levels of students’ sense of self-efficacy between Annenberg and 
demographically similar non-Annenberg schools. The very slight advantages to 
Annenberg schools in 1997 and 1999 were not significant and disappeared by 2001. 

Figure 11. Student Academic Engagement: Average Standardized Change Unit Differences from 1994 
School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 

1994 to 2001 




Figure 12. Student Sense of Self-Efficacy: Average Standardized Change Unit Differences from 1997 
School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 



1997 to 2001 



3 i 



II 

1J 



1 - 



-1 - 



-3 J 




-Annenberg Schools 



— O -Non-Annenberg Schools 



1997 



1999 



2001 



Student Classroom Behavior. Student classroom behavior in Annenberg schools 
declined between 1994 and 2001 at a small but steady rate, although it stayed within 
the “moderately positive” category (see Figure 13). In 2001, students in Annenberg 
schools were somewhat less inclined than in 1994 to respect each other, work well 
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together, and help each other learn. They were somewhat less likely to report that 
students who do well in school are not made fun of; that students work together to 
solve problems; and that they get along well, care about each other, and treat each 
other with respect. They were also somewhat more likely to indicate that students 
look out just for themselves and like to put others down. They were mixed in their 
reports that students do not disrupt class, however. Annenberg schools were no 
different from demographically similar non-Annenberg schools on this measure. 

Figure 13. Student Classroom Behavior: Average Standardized Change Unit Differences from 1994 
School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 

1994 to 2001 
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Student Social Competence. In 1997, student social competence among 
Annenberg schools could be described as “moderate” (see Figure 14). That is, 
students were likely to report that they were good at helping people, taking turns, 
working with other students, listening carefully to what others say, and found it easy 
to make suggestions without being bossy. Students were mixed in their reports that 
they could always find a way to help others end arguments. Levels on this measure 
declined slightly by 2001, which mirrored a decline in demographically similar non- 
Annenberg schools. There were no statistically significant differences between these 
groups of schools on this outcome. 
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Figure 14. Student Social Competence: Average Standardized Change Unit Differences from 1997 
School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 



1997 to 2001 
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School Development 

Although there was little improvement and no overall differences in student 
academic and non-academic outcomes between Annenberg and demographically 
similar non-Annenberg schools, it is nonetheless important to examine trends in 
school development. Following the Challenge’s logic and the logic inherent in the 
Model of Essential Supports, one would expect that before improvement in student 
outcomes can occur, schools need to develop in ways that would promote that 
improvement. Therefore, it is important to see whether Annenberg schools as a 
group developed in ways that would lay such a foundation. 

This section presents findings concerning the development of Annenberg schools 
on each of the Essential Supports. In addition to our analyses of trends in the survey 
data and comparisons between Annenberg and demographically similar non- 
Annenberg schools, we provide examples of change in each Support from our field 
research to illustrate the overall patterns of development across Annenberg schools. 

First are findings about the development of instruction and student learning 
climate, the two supports most proximal to student learning in school. Next are 
findings about the development of school leadership, teacher professional 
community, and parent and community involvement, those supports that provide 
the organizational foundation for teaching and learning. We conclude with findings 
concerning relational trust and instructional program coherence, the two overarching 
Supports. 
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In preview, analyses indicate that as a group, Annenberg schools improved on 
almost one-half of the Essential Supports. Development was particularly strong in 
some measures of instruction and student learning climate. There was a consistent 
pattern among school leadership, teacher professional community, and professional 
development — most development on the measures of these Supports occurred 
between the baseline year and 1999, but much of this progress eroded by 2001. 
Overall, there was almost no difference in patterns of development between 
Annenberg and demographically similar non-Annenberg schools. In 1999, there were 
small differences that favored Annenberg schools but, by 2001, these disappeared 
with virtually no exception. At the end of the Challenge, there were essentially no 
statistically significant differences between Annenberg and non-Annenberg schools 
on measures of the Essential Supports. 

It should be noted that the analyses revealed some very large standardized change 
unit differences in several of the Essential Supports from year to year, including some 
of more than two standard deviations. Although these findings may seem quite 
improbable, several factors should be considered when interpreting them. Measures 
of the Essential Supports are constructed on 10-point scales and the distribution of 
responses on these scales is often very narrow. On a 10-point scale, the standard 
deviations of measures at the school level are relatively small, indicating little 
variation across schools (indeed most variations in these measures are within schools). 
Taking into account the range of the scales and the size of the standard deviations, a 
standardized change unit difference of two standard deviations may represent only a 
one-point or 10 percent difference on a particular measure. A 10 percent difference 
might be a very reasonable amount of change to occur during a seven-year period. In 
addition, when one considers the substantive categories that define a measure’s 
different levels, a one-point difference on a 10-point scale may mean relatively small 
movement within a category (e.g., “limited”), but not movement from one category 
to another (e.g., from “limited” to “moderate”). 



High-Quality Classroom Instruction 

The Model of Essential Supports defines high-quality classroom instruction by three 
basic elements. 29 The first is student exposure to subject matter. In high-quality 
instruction, subject matter is introduced at a steady, challenging pace and 
coordinated within and across grade levels. Teachers may teach basic skills, but they 
seldom rely on repetition and review. They introduce new and more intellectually 



29 See Delpit (1998); Elmore and Burney (1997); Good and Brophy (1997); Newmann and Associates 
(1996); and Smith, Lee, and Newmann (2001). 
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rigorous concepts in a manner that is appropriately challenging. The second is how 
teachers engage students in subject matter, or the intellectual demands they make in 
the classroom. In high-quality instruction, teachers make frequent use of 
intellectually challenging assignments that require students to study a topic in depth, 
produce new knowledge and understanding, communicate and explain to others 
what they have learned, and draw connections to problems and situations beyond 
school. The third element concerns the instructional methods teachers use to engage 
their students in intellectually demanding ways. The Model of Essential Supports 
focuses on two types of instructional methods. The first, didactic instruction, refers 
to the use of whole-class presentation, recitation, and individual student work to 
transmit and promote the acquisition of specific knowledge and skills. The second, 
interactive instruction, refers to the use of interactive, problem-oriented, 
differentiated strategies to promote analysis, application, and production of 
knowledge. A combination of the two, with a relatively strong emphasis on 
interactive practices, characterizes high-quality instruction. Finally, high-quality 
instruction is supported by adequate time for teaching and learning and by strong 
curricular and instructional materials. 

Low-quality instruction is characterized by slow introduction of new subject 
matter; frequent repetition, review, and reteaching; and lack of coordination within 
and across grade levels. Teachers rarely expose their students to intellectually 
challenging subject matter and require little more than the acquisition of discrete 
pieces of knowledge and skills. Students engage subject matter superficially and are 
not often asked to apply, analyze, or evaluate it. Students are not required to 
communicate, explain, or support their work, or to connect it to a problem or 
situation beyond school. Teachers rely primarily on didactic teaching methods and 
make little use of interactive instruction. Curricular and instructional materials are 
weak. Instructional time is not well preserved, nor is it used to full advantage. 



Development across Annenberg Schools 

We examined four measures associated with these elements of high-quality 
instruction: (a) demand for authentic intellectual work; (b) teachers’ emphasis; (c) 
use of interactive instructional practices; and (d) use of didactic instructional 
practices. The first measure assesses the challenge with which teachers engage 
students. The second focuses on student work with subject matter through writing. 
The last two assess teachers’ use of different types of instructional methods to engage 
students in intellectually demanding ways. Overall, between 1997 and 2001, 
instruction in Annenberg schools improved on three of the four measures (see Table 
4). Teachers’ demand for intellectual work, emphasis on writing, and use of 
interactive practices were all greater in 2001 than in 1997. Teachers’ use of didactic 
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practices did not increase. On all but teachers’ use of didactic practices, there were no 
statistically significant differences between Annenberg and demographically similar 
non-Annenberg schools. Teachers’ use of didactic practices was lower in Annenberg 
schools in both 1997 and 2001. 



Table 4. Development of Instruction in Chicago Annenberg Schools, 1997 to 2001: 
Summary of Findings 





1994 OR 
1997 


2001 


DIFFERENCE 
IN MEANS 


STANDARDIZED 
CHANGE UNIT 
DIFFERENCE 


NON- 

ANNENBERG 

COMPARISON 


Improved 


Demand for 
Authentic 
Intellectual Work 


Low 


High 


+ 0.33 


+2.54 




Writing 

Emphasis 


Moderate 


Fairly 

intensive 


+ 0.89 


+ 2.28 


E 


Interactive 

Instruction 


Regularly 


Regularly 


+ 0.39 


+ 2.60 




No Net Change 


Didactic 

Instruction 


Infrequent 


Infrequent 


+ 0.04 


+ 0.17 


_ 



Note: Measures are considered improved or weakened if the difference in means change between 1994 or 1997 and 2001 is 
statistically significant at or beyond 0.01 {p < 0.01). Comparisons to non-Annenberg schools are for 2001. A V’ indicates that 
Annenberg schools were stronger than non-Annenberg schools on a particular measure (/> < 0.01). A “a” indicates that the two 
groups of schools were statistically equivalent. A indicates that Annenberg schools were weaker than non-Annenberg schools 
on the measure {p < 0.01). 



Demand for Authentic Intellectual Work. Demand for authentic intellectual 
work rose steadily among Annenberg schools between 1997 and 2001 (see Figure 
1 5). It rose from the high end of the “low” category to the low end of the “high” 
category of the measure. This means that in 1997, on average, teachers in Annenberg 
schools asked students to elaborate their ideas and organize and synthesize 
information less than once a week; spent between 5 percent and 35 percent of their 
class time synthesizing ideas from reading, differentiating fact from opinion, and 
drawing inferences; and more than 50 percent of their time analyzing or interpreting 
literature. On average, between 10 and 50 percent of teachers’ lessons in Annenberg 
schools dealt with a topic in-depth and asked students to produce original work. In 
2001 , on average, teachers in Annenberg schools asked students to elaborate their 
ideas and organize and synthesize information once or twice a week; spent between 
35 and 50 percent of their class time on synthesizing ideas from reading, 
differentiating fact from opinion, and drawing inferences; and between 50 and 75 
percent of their time analyzing and interpreting literature. On average, between 50 
and 75 percent of lessons dealt with studying a topic in depth and having students 
produce original work. There were no statistically significant differences between 
Annenberg and demographically similar non-Annenberg schools on this measure. 
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Figure 15 . Demand for Authentic Intellectual Work: Average Standardized Change Unit Differences 
from 1997 School System Mean for Annenberg and Demographically Similar Non-Annenberg 
Schools, 1997 to 2001 
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Writing Emphasis. In Annenberg schools, teachers’ emphasis on writing rose 
slightly between 1997 and 1999 and then rose substantially between 1999 and 2001 
(see Figure 16). This increase was coincident with the introduction of the new 
Illinois Standards Achievement Test (ISAT) that contained a new, more intensive 
focus on writing. In 1997, emphasis on writing was “moderate.” Teachers typically 
had students write a one-page assignment once or twice a semester and one or two 
paragraphs once or twice a week. They did not typically ask students to write 
anything longer, but did have students revise and edit their writing once or twice a 
month. By 2001, emphasis on writing was “fairly intensive.” Teachers were more 
likely to have students write one to two paragraphs nearly every day, one page once 
or twice a month, and one to three pages once or twice a semester. They were more 
likely to have students revise and edit their written work once or twice a week. There 
were no statistically significant differences between Annenberg and demographically 
similar non-Annenberg schools. 

Interactive Instruction. Teachers in Annenberg schools increased their use of 
interactive teaching methods between 1997 and 2001 (see Figure 17). Nonetheless, 
their use was “regular” as opposed to “frequent.” By 2001, teachers were somewhat 
more likely to assign projects of one week’s duration to students once or twice a 
month, have students discuss what they read in small groups, and use cooperative 
learning groups at least once or twice a week. They were somewhat more likely to 
consider student participation in class to be very important in their judgment of 
student learning. There were no statistically significant differences between 
Annenberg and non-Annenberg schools on this measure. Although Annenberg 
teachers exhibited slightly greater use of interactive practices in 1999 and 2001, these 
differences were not statistically significant. 
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Didactic Instruction. Finally, in Annenberg schools, teachers’ use of didactic 
practices remained steady and at relatively low levels between 1997 and 2001 (see 
Figure 18). In both 1997 and 2001, Annenberg teachers’ use of didactic instruction 
was “infrequent.” This means that teachers tended to use highly structured call and 
response exercises or had students memorize facts less than once or twice a week. 
They lectured students for more than half a lesson period less than once or twice a 
month, although they may have had students read aloud as often as once or twice a 
week. In 1997 and 2001, Annenberg teachers made significantly less use of didactic 
methods than teachers in demographically similar non-Annenberg schools. In 1999, 
however, they made only somewhat less use of these methods and that difference was 
not statistically significant. 



Figure 16. Writing Emphasis: Average Standardized Change Unit Differences from 1997 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1997 to 2001 




Figure 17. Interactive Instruction: Average Standardized Change Unit Differences from 1997 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1997 to 2001 
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Figure 18. Didactic Instruction: Average Standardized Change Unit Differences from 1997 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1997 to 2001 
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Examples from the Field 

As noted in an earlier Chicago Annenberg Research Project report, the most 
prevalent school-level change in instruction we observed in our field research was an 
increased effort to prepare students for standardized tests. 30 In 2001, all but two of 
the 12 schools in our sample were spending greater amounts of time and effort 
teaching students how to take timed multiple-choice exams. We did observe teachers 
in some schools working individually or in small groups to increase their use of 
interactive teaching strategies, raise the intellectual demand of their assignments, and 
coordinate subject matter among their classrooms. Unfortunately, there were very 
few examples of schoolwide efforts to improve instruction in these ways. 

Rigoberta Menchu Elementary School experienced many of the instructional 
trends that are present in the survey data. 31 For several years prior to the Challenge, 
Menchu was working with its Annenberg External Partner to implement a 
comprehensive literacy program to improve classroom instruction and student 
achievement. Although it differed somewhat at the primary and intermediate grade 
levels, this program helped teachers develop instructional strategies that called for a 
balance between skills practice, literature-based activities, writing across the 
curriculum, and addressing multiple learning styles. The External Partner and most 
teachers and administrators at Menchu agreed that these strategies would strengthen 
the overall quality of instruction at the school. 

Classroom observations, instructional artifacts, and interviews indicate that 
instruction improved at Menchu between 1996 and 1999. Teachers began to make 
subject matter and instructional assignments more intellectually challenging for 



30 Wenzel et al. (2001). 

31 Pseudonyms are used to maintain the anonymiry of the field research sites. 
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students. They worked to strengthen the link between classroom instruction and 
students’ experiences outside of school. They introduced new content at a faster pace 
and reduced the amount of review and repetition. Teachers also increased their use of 
interactive teaching methods by introducing literature circles and small group 
collaborative writing projects. During the 1997-98 school year, Menchu’s External 
Partner introduced student assessments tailored to the new literacy curriculum that 
would help teachers do a better job identifying individual student’s learning needs. In 
interviews, teachers reported that they incorporated more and more elements of the 
literacy program into their everyday teaching. Classroom observations corroborated 
these reports. 

Instructional improvement can be fragile, however, and Menchu’s experience 
illustrates this. In 2001, while some teachers were still teaching in ways that were 
consistent with the new literacy program, the schoolwide improvement in instruction 
that occurred between 1996 and 1999 had eroded. Menchu lost to retirement and 
job transfers a substantial number of teachers who were among the strongest 
implementers of the literacy initiative. Their replacements were unfamiliar with the 
program and, on the whole, did not use the instructional practices it promoted. 
Moreover, the influx of new teachers coupled with reductions in the literacy 
program’s funding made it increasingly difficult for the school’s curriculum 
coordinators to provide adequate professional development and implementation 
support. The External Partner had to reduce the time it spent at Menchu and 
became less available to teachers. With less support and increased demands from a 
growing number of high-need students in their classrooms, Menchu teachers found it 
increasingly difficult to experiment with and implement new instructional strategies. 
At the end of the field research, instruction at Menchu looked much the same as it 
did in 1996. 



Supportive Student Learning Climate 

A strong, supportive student learning climate is characterized by a number of factors 
that include high expectations and press for student achievement and strong social 
support for learning from teachers, parents, and peers. 32 In a strong, supportive 
learning climate, students feel their teachers know them personally and care about 
them as individuals. They count on teachers to notice if they are having academic or 
personal problems and give extra help. Students feel that their peers think school and 



32 See Bryk, Lee, and Holland (1993); Carnegie Council on Adolescent Development (1989); 
Coleman (1988); Dorsch (1998); King and Mathers (1997); Lee et al. (1999); Marks, Doane, and 
Secada (1996); McDill, Natriello, and Pallas (1986); Noddings (1998); Raudenbush (1984); Sebring 
et al. (1996); Shouse (1996); and Sizer (1984 and 1992). 
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learning are important. They have a sense of being physically and psychologically safe 
in their schools and classrooms. There are few disciplinary problems and those that 
occur are handled firmly and fairly. Teachers and students treat each other with 
respect and trust. A strong student learning climate is supported by efforts to develop 
and sustain a schoolwide focus on teaching and learning and optimize instructional 
time. 

Weak student learning climates lack focus on academic learning. Students are not 
pressed toward high achievement and they receive little support from teachers, 
parents, and peers. Students do not necessarily feel that their teachers know them 
personally or care about them as individuals. They may not feel that they can trust 
their teachers to be fair or notice when they have problems. In weak learning 
climates, students may not feel physically or psychologically safe. Instructional time 
may be interrupted frequently and discipline problems may detract from teaching 
and student learning. 



Development across Anncnbcrg Schools 

Four measures were used to map the development of student learning climate across 
Annenberg schools: (a) classroom personalism; (b) safety; (c) press toward academic 
achievement; and (d) peer support for academic work. Overall, between 1994 and 
2001, Annenberg schools as a group improved on two indicators of learning climate: 
classroom personalism and school safety (see Table 5). These were among the 
strongest areas of development across all of the Essential Supports. At the same time, 
peer support for academic work declined across Annenberg schools and levels of press 
toward academic achievement were the same in 2001 as in 1994. In all but one 
measure of classroom instruction, there were no statistically significant differences 
between Annenberg and demographically similar non-Annenberg schools on these 
measures of learning climate. 
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Table 5. Development of Student-Centered Learning Climate in Chicago Annenberg Schools, 
1994 or 1997 to 2001, Summary of Findings 





1994 OR 1997 


2001 


DIFFERENCE 
IN MEANS 


STANDARDIZED 
CHANGE UNIT 
DIFFERENCE 


NON- 

ANNENBERG 

COMPARISON 


Improved 


Classroom 

Personalism 


Considerable 


Considerable 


+ 0.84 


+ 3.23 




Safety 


Somewhat 

safe 


Mostly safe 


+ 1.10 


+ 1.90 


e 


No Net Change 


Press toward 

Academic 

Achievement 


Moderate 


Moderate 


+ 0.03 


0.14 




Weakened 


Peer Support 
for Academic 
Work 


Moderate 


Moderate 


-0.37 


-1.19 





Note: Measures are considered improved or weakened if the difference in means between 1994 or 1997 and 2001 is statistically 
significant at or beyond 0.01 {p < 0.01). Comparisons to non-Annenberg schools are for 2001. A “+” indicates that Annenberg 
schools were stronger than non-Annenberg schools on a particular measure {p < 0.01). A “e” indicates that the two groups of 
schools were statistically equivalent. A indicates that Annenberg schools were weaker than non-Annenberg schools on the 
measure (p < 0.01). 



Classroom Personalism. Students’ perceptions of the care, concern, and 
attention they received from their teachers were stronger in 2001 than in 1994 (see 
Figure 19). In 1994, students reported “considerable” but not “strong” levels of 
personalism. That is, they agreed or strongly agreed that their teachers believed they 
could do well in school. They agreed but did not strongly agree that their teachers 
were willing to give extra help, noticed if they were having trouble learning 
something, helped them catch up if they fell behind, and really listened to what they 
had to say. Students were mixed in whether they agreed or disagreed that their 
teachers related subject matter to their personal interests. In 2001, students’ reports 
of personalism were higher in the “considerable” category of the measure. There were 
no statistically significant differences between Annenberg and demographically 
similar non-Annenberg schools on this measure. 

Safety. Students’ sense of safety in and around Annenberg schools rose between 
1994 and 1997 and remained relatively constant between 1997 and 2001 (see Figure 
20). In 1994, students considered Annenberg schools “somewhat safe.” By 2001, 
they considered them “mostly safe.” This means that in 1994, students felt only 
somewhat or mostly safe in their classrooms, in the hallways and bathrooms of their 
schools, and traveling between home and school, and they felt only somewhat safe in 
the area around their schools. In 2001, students were more likely to report that they 
felt very safe in their classrooms and mostly or very safe elsewhere in their schools, in 
the area around their schools, and traveling between home and school. There were no 
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differences between Annenberg and demographically similar non-Annenberg schools 
on this measure. 

Press toward Academic Achievement. Unlike classroom personalism and school 
safety, press toward academic achievement in Annenberg schools was much the same 
in 2001 as it was in 1994 (see Figure 21). It declined between 1997 and 1999, but 
then increased between 1999 and 2001 to roughly 1994 levels. In both 1994 and 
2001, students reported experiencing “moderate” as opposed to “high” levels of press. 
This means that students agreed, although not always strongly, that their teachers 
expected them to do well in school, praised them when they worked hard, did not 
think they were dumb if they asked about things they did not understand, and 
expected them to finish their homework and do extra work. There were no 
statistically significant differences between Annenberg and demographically similar 
non-Annenberg schools on this measure. 

Peer Support for Academic Work. Finally, in contrast to the trend in classroom 
personalism, peer support for academic work declined steadily in Annenberg schools 
between 1994 and 1999, and then leveled out between 1999 and 2001 (see Figure 
22). Between 1994 and 2001, peer support fell from the high end of the “moderate” 
category of the measure to the low end. Moderate peer support means that students 
report that most but not all of their peers try hard to get good grades, attend all of 
their classes, pay attention in class, and think homework is important. There were no 
statistically significant differences between Annenberg and non-Annenberg schools 
on this measure. 

Figure 19. Classroom Personalism: Average Standardized Change Unit Differences from the 1994 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1994 to 2001 
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Figure 20. Safety: Average Standardized Change Unit Differences from 1994 School System Mean for 
Annenberg and Demographically Similar Non-Annenberg Schools, 1994 to 2001 
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Figure 21. Press toward Academic Achievement: Average Standardized Change Unit Differences from 
1994 School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 
1994 to 2001 
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Figure 22. Peer Support for Academic Work: Average Standardized Change Unit Differences from 
1994 School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 
1994 to 2001 




Examples from the Field 

An earlier Annenberg research project report described several field research sites that 
made substantial efforts to improve their physical environments to foster more 
supportive student learning climates. 33 At other sites, school staff worked to increase 
safety and reduce behavioral problems, tried to develop more personalized 
relationships between students and adults, and provided greater recognition of 
student work and academic success. Between 1999 and 2001, many of the field 
research schools continued to work to improve student learning climate. Some found 
that newly hired teachers brought renewed enthusiasm for teaching and learning. 
Others took advantage of CPS-funded capital improvements and rearranged 
classrooms and other learning spaces to promote communication among teachers and 
create environments that were more conducive for teaching and learning. At the same 
time, other field research schools did little to improve their learning climates. They 
remained disorganized and chaotic places where neither teachers nor students felt 
well supported in their work. These schools failed to make even the most basic 
changes in their physical facilities or scheduling to improve their learning climates. 

Oscar Arias Sanchez Elementary School made substantial strides in developing a 
stronger, more supportive student learning climate. Building repairs and renovations 
dramatically improved space and aesthetics. In 1997, Sanchez suffered from severe 
overcrowding. Every available space was used for instruction — classes were held in 



33 Wenzel et al. (2001). 
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the cafeteria, gymnasium, and even closets. Science, art, and other special classes had 
to move from room to room throughout the day. Moreover, the school was 
disorganized and noisy and the daily schedule often changed at the last minute. In 
1999, an addition to the building created much needed instructional space. Teachers 
received permanent classrooms and the noise level was greatly reduced. Moreover, a 
new custodial staff spruced up the appearance of the building. Walls were freshly 
painted and teachers began to decorate the halls and classrooms with student work. 
With order established in the hallways, teachers and the principal turned their 
attention to protecting instructional time from interruptions. Indeed, after the 
addition was completed, both teachers and students were observed to be more 
enthusiastic and invested in the school. They were better able to focus on teaching 
and learning. 

Between 1999 and 2001, the principal and teachers at Sanchez continued their 
work to improve the school’s learning climate. Building space was reorganized to 
place teachers of the same grade level closer to each other. With increased 
opportunities for interaction, teachers reported that it was easier to get to know their 
students personally. They also reported greater opportunity to learn about school 
resources to help struggling students succeed. In addition, more frequent 
communication between teachers and administrators helped Sanchez adapt its 
instructional program and support services to improve student conduct and learning. 

Strong School Leadership 

According to the Model of Essential Supports, strong school leadership is based on a 
clear mission and vision for the school. 3 * It is broadly based and inclusive. It involves 
the principal, faculty and staff, parents, and LSC members. The principal and other 
administrators communicate well with teachers and involve them in school-level 
decision making. Teachers work with colleagues and administrators to formulate 
plans for school development, particularly those related to instructional 
improvement. The principal takes an active role in instruction and its development 
by recruiting and retaining effective staff members; encouraging teacher professional 
development, experimentation, and innovation; and reducing classroom 
interruption. Strong leadership communicates effectively with the school 
community. It is strategic and accepts responsibility for fair enforcement of policies, 
program implementation, and for enacting the school’s vision of the future. School 
management is efficient and effective. 

On the other hand, consolidated principal power and authoritarian decision 
making characterize weak school leadership. It fails to articulate a clear vision for the 
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school and does litde to communicate goals and plans for development. It does not 
focus on instruction and there is little accountability. School management is chaotic 
and unpredictable. The principal fails to support teachers, neither helping them in 
their professional development nor protecting them from interruptions to their work. 



Development across Annenberg Schools 

Four measures were used to map changes in school leadership across Annenberg 
schools: (a) inclusive leadership; (b) teacher influence in decision making; (c) joint 
problem solving; and (d) principal instructional leadership. Overall, teacher influence 
in decision making increased in Annenberg schools between 1994 and 2001 (see 
Table 6). At the same time, levels of inclusive leadership, which includes parent and 
community involvement, declined. Principal instructional leadership and levels of 
joint problem solving were much the same in Annenberg schools in 2001 as in 1994. 
As described in more detail below, a few statistically significant differences were 
found between Annenberg and demographically similar non-Annenberg schools on 
several measures of school leadership, but only in 1997 and 1999. These initial 
improvements in Annenberg schools disappeared after 1999. In 2001, there were no 
differences between Annenberg and demographically similar non-Annenberg schools 
on any dimension of school leadership. 



Table 6. Development of Leadership in Chicago Annenberg Schools, 1994 to 2001: 
Summary of Findings 





1994 


2001 


DIFFERENCE 
IN MEANS 


STANDARDIZED 
CHANGE UNIT 
DIFFERENCE 


NON- 

ANNENBERC 

COMPARISON 


Improved 


Teacher Influence in 
Decision Making 


Moderate 


Moderate 


+ 0.21 


+ 0.33 


s 


No Net Change 


Principal Instructional 
Leadership 


Strong 


Strong 


-0.04 


- 0.04 




Joint Problem Solving 


Strong 


Strong 


-0.14 


-0.15 


g 


Weakened 


Inclusive Leadership 


Positive 


Positive 


- 0.29 


- 0.34 





Note: Measures are considered improved or weakened if the difference in means between 1994 or 1997 and 2001 is statistically 
significant at or beyond 0.01 ip < 0.01). Comparisons to non-Annenberg schools are for 2001. A indicates that Annenberg 
schools were stronger than non-Annenberg schools on a particular measure ip < 0.01). A “e” indicates that the two groups of 
schools were statistically equivalent. A indicates that Annenberg schools were weaker than non-Annenberg schools on the 
measure ip < 0.01). 



34 See Blumbergand Greenfield (1980); Bryk et al. (1998a); Chubb and Moe (1990); Lightfoot 
(1983); Lipsitz (1984); Newmann and Wehlage (1993); and Sebring and Bryk (2000). 
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Inclusive Leadership. In 1994, Annenberg schools as a whole had “positive” 
levels of inclusive leadership. That is, teachers across Annenberg schools were likely 
to agree or strongly agree that their principals promoted parent and community 
involvement in school and they tended to agree, though not strongly, that their 
principals worked to create a sense of community in their schools and were 
committed to shared decision making. Levels of inclusive leadership in Annenberg 
schools declined slightly from 1994 to 1997, rose between 1997 and 1999, but fell 
again after 1999 (see Figure 23). By 2001, inclusive leadership was lower across 
Annenberg schools than in 1994 although it remained within the “positive” category 
of the measure. While inclusive leadership was greater in Annenberg schools than in 
demographically similar non-Annenberg schools in 1999, there were no statistically 
significant differences between the two groups by 2001. 

Teacher Influence in Decision Making. This was the only measure of 
leadership that was stronger among Annenberg schools in 2001 than in 1994. This 
difference existed despite losses between 1999 and 2001 of initial improvement that 
occurred between 1994 and 1999 (see Figure 24). In 1994, the level of teacher 
influence in Annenberg schools was “moderate.” Teachers reported that they had 
some or a great deal of influence in determining instructional materials for their 
classes. They tended to agree that they were comfortable voicing their concerns and 
were involved in making important decisions at their schools. They reported having 
some influence over establishing curricular programs and setting standards for 
student behavior, but they reported having a little or only some influence over their 
teaching assignments, their schools 5 use of discretionary funds, and the hiring of 
principals and other school personnel. In 2001, teacher influence was slightly greater 
than influence in 1994 but still remained “moderate.” While teacher influence was 
stronger in Annenberg schools than in non-Annenberg schools in 1997 and 1999, 
there were no statistically significant differences between these groups of schools on 
this measure in 2001. 
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Figure 23. Inclusive Leadership: Average Standardized Change Unir Differences from 1994 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1994 to 2001 
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Joint Problem Solving. In 1997, joint problem solving in Annenberg schools 
could be described as “strong.” Overall, teachers tended to agree that other teachers 
in their schools did not dismiss or ignore problems, did a good job talking through 
differences in opinions, and expressed their personal views openly. Also, teachers 
tended to agree that their schools have good processes for resolving conflicts and use 
faculty meetings for problem solving. Levels on this measure in Annenberg schools 
held steady between 1997 and 2001 with fluctuations that were not statistically 
significant (see Figure 25). Overall, joint problem solving in Annenberg schools in 
2001 was much the same as it was in 1997. It did not rise to “very strong” where 
teachers would be more likely to strongly agree that these practices existed in their 
schools. Differences between Annenberg and demographically similar non- 
Annenberg schools were statistically significant in 1997 and 1999 but not in 2001. 



Figure 24. Teacher Influence in Decision Making: Average Effect Size Differences from 1994 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1994 to 2001 
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Figure 25. Joint Problem Solving: Average Standardized Change Unit Differences from 1997 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1997 to 2001 
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Principal Instructional Leadership. In 1994, instructional leadership in 
Annenberg schools was “strong” according to the measure. Teachers were likely to 
agree but not strongly agree that their principals made expectations for teaching 
clear; set high standards for both teaching and student learning; communicated a 
clear vision for the school; pressed them to implement what they learned in 
professional development activities; understood how students learn; and tracked 
student academic progress. Instructional leadership in Annenberg schools rose 
slightly from 1997 to 1999 but fell by 2001 to about its 1994 levels (see Figure 26). 
The increase between 1997 and 1999 and the decline between 1999 and 2001 were 
statistically significant, but levels of instructional leadership in 2001 and 1994 were 
statistically equivalent; both were within the “strong” category. There were no 
statistically significant differences between Annenberg and demographically similar 
non-Annenberg schools on this measure in any year. 



Figure 26. Principal Instructional Leadership: Average Standardized Change Unit Differences from 
1994 School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 
1994 to 2001 
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Examples from the Field 

Field research conducted between 1996 and 1999 found many examples of 
improvement in school leadership. Increases in teacher participation in grade- and 
school-level decision making were documented. So too were greater emphases on 
instructional improvement in school-level planning and decision making. In a 
number of sites, teachers became more involved in school- and grade-level program 
development, especially in establishing goals for more intellectually ambitious 
teaching and learning. A number of principals worked to involve parents and other 
members of their school communities in school leadership. 

By 2001 , several field research schools built upon or sustained earlier 
improvement in leadership. Others failed to develop at all. In a third group, 
leadership that was initially improving had become problematic. For example, in one 
school, a new principal reversed the progress the previous administration had made. 

In several others, tension arose between teachers and principals over participation in 
and control over decision making. In these schools, teachers’ expectations for 
involvement began to conflict with the principals’ sense of accountability for school 
performance and their belief that they needed to take back control of some decisions. 

The example of Renee Cassin Elementary School shows how strong principal 
leadership supported school development in some ways and undermined it in others. 
In 1997, Cassin was under the threat of academic probation because of its 
persistently low ITBS scores. When the new principal was hired that year, he took a 
number of decisive actions. He reviewed all programs and outside organizations 
operating at the school, eliminated those that served only a few students or did not 
focus on improving classroom instruction, and retained those that provided 
professional development and instructional support. He obtained new instructional 
materials for teachers and counseled teachers he felt were ineffective to move to a 
different school. He also worked to create stronger relationships between parents and 
teachers. 

In 1999, teachers at Cassin reported that the principal encouraged them to 
participate in professional development. He provided money and time for them to 
attend local and national conferences. He restructured the school day so that the 
whole staff could meet on Friday afternoons twice a month and established two 
common planning periods each week for grade-level meetings. Teachers believed that 
these efforts helped strengthen instructional program coherence at Cassin and 
develop the school’s professional community. In 2001, however, several weaknesses 
in the principal’s leadership had begun to undermine much of the progress that had 
been made. Teachers had become less supportive of the principal, characterizing his 
leadership as authoritative and even authoritarian. Although he spoke about 
involving teachers in decision making and school improvement planning, he 
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admitted that he made most of the important decisions at the school. The direct and 
consolidated nature of his leadership and fallout from several contentious decisions 
had begun to frustrate the faculty and had led several of the school’s more productive 
teachers to resign or transfer to other schools. 



Strong Professional Community 

Teacher professional community refers to the quality of working relationships among 
teachers and other staff members at a school and the social and normative resources 
these relationships provide. In strong professional communities, teachers have a clear 
and common vision for the future and a shared sense of the school’s mission and 
goals. 35 They have a common language and similar beliefs and values. Teachers are 
deeply committed to high-quality instruction; they share responsibility and 
accountability for their students’ success and for achieving the school’s goals. 

Teachers in strong professional communities are highly collaborative. They exchange 
information about what they have learned from professional experience and research 
and engage in reflective conversation about their own practices and assumptions. In 
strong professional communities, there is a clear disposition toward ongoing learning 
and innovation. Members do not always agree on everything, but because of high 
levels of trust, disagreement is most often constructive rather than destructive. 

In weak professional communities, teachers work in relative isolation from one 
another. They may be cordial and interact socially, but they rarely share information, 
discuss problems, or collaborate. Teachers in weak professional communities do not 
feel accountable to colleagues or to the school as a whole. They do not share a vision 
for the future nor do they agree on a set of goals for school development. They lack a 
common language and are guided by norms of autonomy and privacy. 

Disagreements are rarely channeled in productive directions. At best, they remain 
unresolved in a state of detente with teachers agreeing to disagree. 



Development across Annenberg Schools 

Six measures were used to trace changes in teacher professional community in 
Annenberg schools: (a) peer collaboration; (b) focus on student learning; (c) 
orientation toward innovation; (d) collective responsibility; (e) reflective dialogue; 
and (f) teacher commitment to school. Overall, in 1994, teacher professional 



35 Bryk ct al. (1998a); Darling Hammond (1990); DuFourand Eakcr (1998); Licbcrman (1995); 
Little (1999); Louis, Kruse, and Associates (1995); Newmann and Wehlage (1995); and Rosenholtz 
(1989). 
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community in Annenberg schools could be described as reasonably strong, with the 
exception of teacher orientation toward innovation (see Table 7). 



Table 7. Development of Teacher Professional Community in Chicago Annenberg Schools, 
1 994 to 2001 : Summary of Findings 





1994 


2001 


DIFFERENCE IN 
MEANS 


STANDARDIZED 
CHANGE UNIT 
DIFFERENCE 


NON- 

ANNENBERG 

COMPARISON 


Improved 


Peer 

Collaboration 


Significant 


Significant 


+ 0.32 


+ 0.34 


s 


Focus on 

Student 

Learning 


Focused 


Focused 


+ 0.11 


+ 0.13 




Orientation 

toward 

Innovation 


Limited 


Limited 


+ 0.09 


+ 0.11 




No Net Change 


Collective 

Responsibility 


Strong 


Strong 


0.00 


0.00 




Reflective 

Dialogue 


Regularly 


Regularly 


+ 0.02 


+ 0.05 




Weakened 


Teacher 
Commitment 
to School 


Strong 


Strong 


-0.34 


- 0.33 





Note: Measures are considered improved or weakened if the difference in means change between 1994 or 1997 and 2001 is 
statistically significant at or beyond 0.0 1 ip < 0.0 1 ). Comparisons to non-Annenberg schools are for 2001 . A “+” indicates that 
Annenberg schools were stronger than non-Annenberg schools on a particular measure ip < 0.01). A “a” indicates that the two 
groups of schools were statistically equivalent. A ” indicates that Annenberg schools were weaker than non-Annenberg schools 
on the measure ip < 0.01). 



Between 1994 and 2001, peer collaboration, focus on student learning, and 
orientation toward innovation improved. Levels of teachers' collective responsibility 
and reflective dialogue were much the same in 2001 as they were in 1994. Finally, 
teacher commitment in Annenberg schools weakened between 1 994 and 2001 . The 
findings are more complicated than these overall differences suggest, however. 
Annenberg schools as a group improved on the majority of these measures between 
1994 and 1999, but most of these initial improvements were lost. In 2001, there 
were no statistically significant differences between Annenberg and demographically 
similar non-Annenberg schools in teacher professional community. 

Peer Collaboration. Levels on this measure in Annenberg schools rose between 
1994 and 1999 but fell between 1999 and 2001 (see Figure 27). In 1994, peer 
collaboration in Annenberg schools was “significant” according to the measure. That 
is, teachers agreed or strongly agreed that other teachers in their schools were cordial. 
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They agreed but did not strongly agree that collaborative efforts made their schools 
run well, that teachers coordinated instruction across grades, and that teachers 
designed the instructional program together. In 2001, while somewhat stronger, peer 
collaboration remained “significant.” It did not rise to the “extensive” category of the 
measure where teachers would more strongly agree that these types of interactions 
occurred in their schools. Between 1994 and 1999, peer collaboration rose at a 
greater rate in Annenberg schools than in demographically similar non-Annenberg 
schools. Between 1999 and 2001, however, levels fell to roughly the same level as for 
non-Annenberg schools. Although the difference between Annenberg and non- 
Annenberg schools approached statistical significance in 1999, it was not significant 
in any year. 

Focus on Student Learning. Like peer collaboration, focus on student learning 
increased slightly between 1994 and 1999 but then declined by 2001 so it was only 
slightly stronger than in 1994 (see Figure 28). In both 1994 and 2001, teachers in 
Annenberg schools were likely to agree but not strongly agree that their schools 
maximized instructional time, set high standards for student academic performance, 
had well-defined learning expectations for students, and made decisions based on 
what was best for students. Like peer collaboration, focus on student learning in 
Annenberg schools appeared to develop somewhat more strongly than in 
demographically similar non-Annenberg schools. And like peer collaboration, any 
advantage that might have been held by Annenberg schools on this measure 
disappeared by 2001. There were no statistically significant differences between 
Annenberg and non-Annenberg schools in any year on this measure. 

Orientation toward Innovation. Levels on this measure in Annenberg schools 
were relatively weaker than peer collaboration and focus on student learning. In 
1994, orientation toward innovation was within the “limited” category of the 
measure. That year, Annenberg teachers reported that only about half of their 
colleagues really tried to improve their teaching. Some agreed while others disagreed 
that teachers at their schools were continually learning, that they were encouraged to 
grow, and that they had a “can do” attitude. They reported that only some tried new 
ideas or took risks to improve their instruction. Levels on this measure in Annenberg 
schools improved slightly between 1994 and 2001 although they weakened thereafter 
and remained “limited” in 2001 (see Figure 29). Orientation toward innovation was 
slightly stronger in Annenberg schools than in demographically similar non- 
Annenberg schools, particularly in 1999; however, these differences were not 
statistically significant. 
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Collective Responsibility and Reflective Dialogue. In 1994, collective 
responsibility in Annenberg schools was considered “fairly strong.” That is, teachers 
reported that most of their colleagues felt responsible to ensure that all students learn, 
that they set high standards for themselves, and that they help students with their 
self-control. Further, teachers reported that about half or most of their peers took 
responsibility for school improvement, helped discipline students, helped each other, 
and felt responsible when students failed. In 1994, reflective dialogue in Annenberg 
schools occurred “regularly” according to our measure. That is, teachers agreed, but 
did not strongly agree, that they talked informally with one another about 
instruction and shared and discussed student work and assumptions about student 
learning. They agreed but did not strongly agree that they had conversations more 
than once or twice a month about how students learn best and how to manage 
student behavior. In addition, they reported having conversations about developing 
new curriculum and school goals between one to three times a month. Neither 
collective responsibility for student learning nor reflective dialogue changed between 
1994 and 2001 among Annenberg schools (see Figures 30 and 31). For Annenberg 
schools, collective responsibility remained “fairly strong” and reflective dialogue 
continued to occur “regularly.” There were no statistically significant differences 
between Annenberg and demographically similar non-Annenberg schools on either 
of these measures in any year. 

Teacher Commitment to School. In 1994, teacher commitment in Annenberg 
schools was “strong.” This means that teachers agreed or strongly agreed that they felt 
loyal to their schools. They agreed but did not strongly agree that they looked 
forward to school each day, that they would recommend the school to other parents, 
and that they would not want to work at another school. Teacher commitment in 
Annenberg schools declined between 1994 and 2001, especially after 1999 (see 
Figure 32). Despite this decline, levels of commitment remained in the “strong” 
category in 2001. Teacher commitment in Annenberg schools rose slightly between 
1994 and 1999 while it declined in demographically similar non-Annenberg schools. 
And it appears that it was slightly lower in non-Annenberg schools in 2001, although 
this difference was not statistically significant. 
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Figure 27. Peer Collaboration: Average Standardized Change Unit Differences from 1994 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1994 to 2001 
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Figure 28. Focus on Student Learning: Average Standardized Change Unit Differences from 1994 
School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 

1994 to 2001 
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Figure 29. Orientation toward Innovation: Average Standardized Change Unit Differences from 1994 
School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 

1994 to 2001 
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Figure 30. Collective Responsibility: Average Standardized Change Unit Differences from 1994 
School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 
1994 to 2001 
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Figure 31. Reflective Dialogue: Average Standardized Change Unit Differences from 1994 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1994 to 2001 
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Figure 32. Teacher Commitment to School: Average Standardized Change Unit Differences from 
1994 School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 
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Examples from the Field 

Between 1997 and 1999, more field research schools worked to develop teacher 
professional community than any other Essential Support and more succeeded in this 
area than in any other. In several schools, groups of teachers began to work together 
more closely to analyze their classroom practices and address issues of student 
learning. Growing numbers learned to talk effectively with one another about 
improving instruction and began to develop a shared language to do so. As a result, 
teachers’ exposure to different instructional practices increased. 

Between 1999 and 2001, most field research schools maintained their 
improvements in professional community. One strengthened it further as its teachers 
grew even more experienced in working together. For four of the field research 
schools, however, professional communities weakened and began to fall apart. At 
these schools, this coincided with the emergence of consolidated principal leadership 
and decision making. Teachers grew increasingly frustrated by their principals and by 
their declining involvement and influence. Some stopped meeting altogether, feeling 
less supported in their efforts and less committed to their school and its 
improvement. Such changes are illustrated well in the case of Nelson Mandela 
Elementary School. 

In 1997, teachers at Mandela were quite cordial to one another, although very 
few spent time working together. Even though the principal called whole school 
faculty meetings several times a year, teachers did not meet regularly to discuss their 
work. Beginning in 1997, however, a small group began to work with their 
Annenberg External Partner to increase teacher collaboration and promote teacher 
learning and development. 

By the next school year, there were marked differences in the working 
relationships among teachers who participated in the Annenberg initiative. These 
teachers consistently took advantage of their regularly scheduled common planning 
time and more readily identified themselves as a team. They frequently used their 
time together to share their experiences from their professional development, giving 
short presentations about what they learned at conferences and discussing specific 
pedagogical issues such as literature circles, thematic units, or how to implement 
advisory periods. Other teachers began to emulate their example and started to 
interact in more collaborative and reflective ways. Several who did not work with the 
External Partner expressed an interest in working together more like a school-within- 
a-school. The principal also said that he would like departments to function more 
like teams. As one school administrator observed, “Many of the ideas the Annenberg 
teachers have adopted, the whole school is adopting them.” By 1999, Mandela had 
made considerable progress in developing a strong schoolwide teacher professional 
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community, but by 2001 , it had all but disappeared. The principal never fully 
embraced the Annenberg initiative at the school and he left Mandela for another 
position in 2000. This created a void of administrative support for teacher teaming 
and collaborative work. Moreover, the External Partner’s funding was reduced and its 
presence in the school decreased. In the end, teachers returned to working in cordial 
' isolation from one another. 



Parent and Community Involvement 

In schools with strong parent and community support, parents participate in school 
activities and contribute in significant ways to achieving school goals ; 36 They support 
their children’s learning at home and are viewed as a crucial resource. There is trust 
between parents and the school, which is characterized by mutual respect and 
confidence in each other’s abilities. Schools with strong parent and community 
support aggressively promote that support. Teachers cultivate ties with parents and 
the surrounding community. They visit students’ homes and attend neighborhood 
events. Teachers are knowledgeable about community and cultural issues that 
concern students and their families. 

For schools with weak parent and community support, parent involvement is not 
a priority. Consequently, parents seldom help the school achieve its goals and may 
not support student learning at home. Trust, respect, and confidence between 
parents and the school may be weak. The school is largely disconnected from the 
surrounding community and does not take advantage of the support parents and 
community organizations might provide. 



Development across Annenberg Schools 

Change in parent and community involvement was tracked in terms of six measures: 
(a) teacher outreach to parents; (b) parent involvement in school; (c) teachers’ use of 
community resources; (d) teachers’ ties to the community; (e) teachers’ knowledge of 
student culture; and (f) human and social resources in the community (see Table 8). 
Overall, only two measures of parent and community involvement improved among 
Annenberg schools between 1994 and 2001 — teacher outreach to parents and parent 
involvement in school. There was virtually no difference between the baseline years 
and 2001 in any other measure among Annenberg schools, despite some initial 
improvement in teachers’ use of community resources and human and social 




36 See Clark (1983); Delpit (1998); Epstein (1993); Epstein and Dauber (1991); Furstenberg et al. 
(1999); Lareau (1989); and Tyack (1992). 
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resources in the community. Like most other measures of the Essential Supports, 
there were no statistically significant differences between Annenberg and 
demographically similar non-Annenberg schools in any year. 

Teacher Outreach to Parents. In 1994, teacher outreach to parents was 
“significant.” That is, teachers were likely to agree but not strongly agree that parents 
were greeted warmly when they visited the school, that teachers tried to understand 
parents’ problems, that the principal encouraged teachers to communicate with 
parents, and that the school welcomed parent feedback. Teachers’ reactions were 
mixed on whether their schools worked at communicating with parents about 
advancing the school mission and helping children learn. They were likely to disagree 
that parents were invited into classrooms and that they worked closely with parents. 
In Annenberg schools, teachers’ outreach to parents was greater in 2001 than in 1994 
although it declined between 1999 and 2001, remaining in the “significant” category 
(see Figure 33). Trends in Annenberg schools mirrored those in demographically 
similar non-Annenberg schools; there were no statistically significant differences 
between these groups in any year. 



Table 8. Development of Parent and Community Involvement in Chicago Annenberg Schools, 
1 994 or 1997 to 2001: Summary of Findings 





1994 OR 
1997 


2001 


DIFFERENCE 
IN MEANS 


STANDARDIZED 
CHANGE UNIT ' 
DIFFERENCE 


NON- 

ANNENBERG 

COMPARISON 


Improved 


Teacher 
Outreach to 
Parents 


Significant 


Significant 


+ 0.51 


+ 0.70 




Parent 

Involvement in 
School 


Moderate 


Moderate 


+ 0.18 


+ 0.20 




No Net Change 


Teachers’ Use of 

Community 

Resources 


Occasional 


Occasional 


+ 0.07 


+ 0.20 


ca 


Teachers’ Ties 
to the 

Community 


Slight 


Slight 


-0.08 


-0.13 


eg 


Teachers’ 
Knowledge of 
Student Culture 


Significant 


Significant 


-0.04 


-0.06 


£ 


Human/Social 
Resources in the 
Community 


Some 


Some 


+ 0.03 


+ 0.10 


eg 



Note: Measures are considered improved or weakened if the difference in means between 1994 or 1997 and 2001 is statistically 
significant at or beyond 0.01 ip < 0.01). Comparisons to non-Annenberg schools are for 2001. A “+” indicates that Annenberg 
schools were stronger than non-Annenberg schools on a particular measure {p < 0.01). A “a” indicates that the two groups of 
schools were statistically equivalent. A indicates that Annenberg schools were weaker than non-Annenberg schools on the 
measure (p < 0.01). 
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Parent Involvement in School. In 1994, parent involvement in Annenberg 
schools was “moderate.” Teachers in Annenberg schools were likely to report that 
most or nearly all parents picked up their children’s report cards and attended school 
events and parent-teacher conferences. Teachers were likely to report that some to 
about half of parents attended special schoolwide events and helped raise funds for 
the school. They were likely to report that only some parents volunteered to work in 
classrooms. Parent involvement in Annenberg schools rose gradually between 1994 
and 1999 and then declined between 1999 and 2001, remaining in the measure’s 
“moderate” category (see Figure 34). Still, it was greater in 2001 than in 1994. There 
were no statistically significant differences between Annenberg and demographically 
similar non-Annenberg schools in any year. 

Teachers’ Use of Community Resources. Across Annenberg schools in 1997, 
teachers’ use of community resources in their teaching was “occasional” rather than 
“frequent” or “extensive.” Teachers in Annenberg schools were likely to report that 
they used people and events from the community as an example and told students 
about community agencies only once to four times that school year. They consulted 
with community members to better understand students and collected materials 
from community businesses for class only once or twice. They took students on field 
trips or brought in guest speakers from the community only once, twice, or never 
that year. Teachers’ use of community resources increased in Annenberg schools 
between 1997 and 1999 but then declined between 1999 and 2001, resulting in no 
net change (see Figure 35). Although it appears that Annenberg schools increased at a 
greater rate than non-Annenberg schools, the differences between the two groups were 
not statistically significant in any year. 

Teachers’ Ties to the Community. In Annenberg schools in 1997, teachers’ ties 
to their schools’ communities were “slight.” While teachers were likely to report that 
they had friends who lived in their schools’ communities, they shopped there only 
once or twice a month. They reported that they attended recreational activities in 
their schools’ communities two or three times a month but attended the same 
religious services as their students and visited their students’ homes less than once a 
month. In Annenberg schools, the levels of teachers’ ties to the community did not 
change between 1997 and 2001 and they showed no statistically significant 
difference from those in demographically similar non-Annenberg schools (see Figure 

36). 

Teachers’ Knowledge of Student Culture. In Annenberg schools in 1997, 
teachers’ knowledge of their students’ cultures was “significant.” Teachers were likely 
to report that most of their colleagues at their schools were aware of community 
issues. They were likely to report that about half or most talked with students about 
their lives and cultures and that about half tried to learn about students’ cultural 
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backgrounds. This remained unchanged between 1997 and 2001 (see Figure 37). 
There were no statistically significant differences between Annenberg and non- 
•Annenberg schools on this measure. 

Human and Social Resources in the Community. In 1997, there were “some” 
supportive human and social resources in the communities of students attending 
Annenberg schools. These students were likely to agree or strongly agree that people 
in their neighborhoods cared about what happened there. They tended to agree but 
not strongly agree that the parks were safe for young people to play at during the day, 
that adults in the neighborhood knew who the local children were, and that they 
could look up to the adults in their community. Students were mixed on whether 
adults in their neighborhoods made sure the neighborhood children were safe, that 
they could trust people living in their neighborhood, and that community members 
addressed problems in the neighborhood rather than ignoring them. Levels on this 
measure increased across Annenberg schools between 1997 and 1999 but declined 
between 1999 and 2001, resulting in no net change (see Figure 38). There were no 
statistically significant differences between Annenberg and non-Annenberg schools 
on this measure. 

Figure 33. Teacher Outreach to Parents: Average Standardized Change Unit Differences from 1994 
School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 

1994 to 2001 
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Figure 34. Parent Involvement in School: Average Standardized Change Unit Differences from 1994 
School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 

1994 to 2001 
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Figure 35. Teachers’ Use of Community Resources: Average Standardized Change Unit Differences 
from 1997 School System Mean for Annenberg and Demographically Similar Non-Annenberg 
Schools, 1997 to 2001 




Figure 36. Teachers Ties to the Community: Average Standardized Change Unit Differences from 
1997 School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 
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Figure 37. Teachers’ Knowledge of Student Culture: Average Standardized Change Unit Differences 
from 1997 School System Mean for Annenberg and Demographically Similar Non-Annenberg 
Schools, 1997 to 2001 
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Figure 38. Human and Social Resources in the Community: Average Standardized Change Unit 
Differences from 1997 School System Mean for Annenberg and Demographically Similar Non- 
Annenberg Schools, 1997 to 2001 




Examples from the Field 

Compared to the number of field research schools that improved their leadership and 
teacher professional community, fewer strengthened their relationships with parents 
and the community. There were several, however, that did strengthen these 
relationships by establishing new parent education programs, seeking assistance from 
community organizations, and helping their students gain greater access to 
community services. Rigoberta Menchu Elementary School was one such school. 

With two parent coordinators on staff, an estimated 30 parent volunteers a day, 
and eight active parent groups, Menchu devoted substantial attention to cultivating 
parent and community involvement and support. The school invited parents to 
workshops on a variety of topics from how to help children learn to how to prepare 
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income tax forms. Several parent groups worked on encouraging students and 
parents to read at home. 

Between 1997 and 1999, Menchu increased these efforts. Staff developed new 
strategies to promote parent involvement and support. According to an LSC 
representative, the school helped parents gather materials to create a lending library 
of videotapes and books about parental concerns, gangs, drugs, puberty, and how to 
support children’s academic growth. In 1997, a parent and community coordinator 
worked with the school’s Annenberg External Partner to establish the Parent 
Leadership Circle. This committee increased coordination and reduced overlap in 
work among the different parent groups. As a result, they became better organized 
and more autonomous and one of the coordinators was able to work with parents on 
increasing student attendance. 

Menchu also made substantial efforts to help its students take greater advantage 
of community resources. The school’s staff established relationships with community 
health organizations so that students might receive preventative health care services 
like immunizations and physical examinations more readily. 



Relational Trust 

Relational trust is one of two overarching Supports in the Model of Essential 
Supports. It refers to shared confidence in the abilities and integrity of others, mutual 
respect, and personal regard. Strong relational trust is crucial for school 
development. 37 In schools with strong relational trust, teachers feel that their 
principal respects and supports them, looks out for their welfare, and has confidence 
in their expertise. They, in turn, respect their principals as educators. In high-trust 
schools, teachers and parents respect and support each other. Students feel that their 
teachers care about them, listen to their ideas, and keep their promises. Moreover, 
teachers trust and respect each other, communicate openly, and support colleagues 
who lead development efforts. 

In schools with weak relational trust, members of the school community hold 
little respect for and have little confidence in others. Teachers do not necessarily 
believe that their principal trusts and supports them or looks out for their welfare. 
There is little mutual respect and support among parents and teachers, students and 
teachers, or among teachers themselves. 



37 See Bryk and Schneider (1996, 2002); Sebring et al. (1995); and Smylie and Hart (1999). 
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Development across Annenberg Schools 

Four measures were used to examine the development of relational trust in 
Annenberg schools: (a) teacher-principal trust; (b) teacher-teacher trust; (c) teacher- 
parent trust; and (d) teacher-student trust (see Table 9). Overall, relational trust in 
Annenberg schools strengthened between 1994 and 2001. Only teacher-student trust 
failed to improve. Nonetheless, there were no statistically significant differences 
between Annenberg and demographically similar non-Annenberg schools on any 
measure of trust. 



Table 9. Development of Relational Trust in Chicago Annenberg Schools, 1994 or 1997 to 2001: 
Summary of Findings 





1994 OR 
1997 


2001 


DIFFERENCE 
IN MEANS 


STANDARDIZED 
CHANGE UNIT 
DIFFERENCE 


NON-ANNENBERG 

COMPARISON 


Improved 


Teacher- 
Principal T rust 


Strong 


Strong 


+ 0.11 


+ 0.13 


£ 


Teacher-Teacher 

Trust 


Minimal 


Minimal 


+ 0.25 


+ 0.40 




Teacher-Parent 

Trust 


Minimal 


Strong 


+ 0.17 


+ 0.22 


£ 


No Net Change 


Teacher-Student 

Trust 


Strong 


Strong 


-0.05 


-0.02 


£ 



Note: Measures are considered improved or weakened if the difference in means between 1994 or 1997 and 2001 is statistically 
significant at or beyond 0.01 {p < 0.01). Comparisons to non-Annenberg schools are for 2001 . A “+” indicates that Annenberg 
schools were stronger than non-Annenberg schools on a particular measure {p < 0.0 1). A “es” indicates that the two groups of 
schools were statistically equivalent. A indicates that Annenberg schools were weaker than non-Annenberg schools on the 
measure {p < 0.01). 



Teacher-Principal Trust. In 1994, teacher-principal trust in Annenberg schools 
was “strong.” Teachers in these schools were likely to report that they felt somewhat 
or to a great extent respected by their principals. They were likely to agree but not 
strongly agree that they respected their principals as educators; that their principals 
took an interest in their professional development, had confidence in their expertise, 
that their principals placed students’ needs before their own personal needs, were 
effective managers, and looked out for the welfare of their teachers. Similarly, 
teachers were likely to agree but not strongly agree that they trusted their principals 
and felt they could discuss their worries with them. Teacher-principal trust in 
Annenberg schools rose slightly between 1994 and 2001 but remained in the 
“strong” category of the measure (see Figure 39). This reflected the development of 
teacher-principal trust in non-Annenberg schools. There were no statistically 
significant differences between the two groups on this measure in any year. 
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Teacher-Teacher Trust. There is a similar pattern in the development of 
teacher-teacher trust (see Figure 40). Unlike teacher-principal trust, however, levels 
on this measure in Annenberg schools were “minimal” in 1994. That year, teachers 
were likely to report that they felt respected by only some of the other teachers at 
their schools. They agreed but did not strongly agree that teachers in their schools 
respected colleagues who were experts at their craft, that teachers took the lead in 
school improvement efforts, or that they could discuss their worries with other 
teachers. They were mixed on whether the teachers at their schools trusted each 
other. None to only some reported that teachers in their schools cared about each 
other. Teacher- teacher trust in Annenberg schools increased between 1994 and 2001 
and moved toward the high end of the “minimal” category of the measure. There were 
no statistically significant differences between Annenberg and demographically similar 
non-Annenberg schools on this measure in any year. 

Teacher-Parent Trust. In 1994, teacher-parent trust in Annenberg was 
“minimal.” Teachers were likely to report that they respected and felt respected by 
parents only to some extent. They were likely to agree but not strongly agree that 
talking with parents helped them understand students better. Some agreed while 
others disagreed that there was no conflict between parents and teachers and that 
parents were partners in educating children. Teachers were likely to report that none 
or only some of the parents at their schools supported their teaching efforts and did 
their best to help their children learn. None to some of teachers felt good about the 
overall support they received from parents. Teacher-parent trust in Annenberg 
schools strengthened between 1994 and 1999 but weakened between 1999 and 2001 
(see Figure 41). Still, in 2001, teacher-parent trust in Annenberg schools was slightly 
stronger than in 1994 and the increase was enough to move the average level to the 
low end of the “strong” category of the measure. In 2001, teachers were more likely 
to be positive about the above relationships and more likely to report that greater 
numbers of parents supported their efforts and helped children learn. There were no 
differences between Annenberg and demographically similar non-Annenberg schools 
on this measure in any year. 

Teacher-Student Trust. In 1997, teacher-student trust in Annenberg schools 
was “strong.” Students were likely to agree but not strongly agree that their teachers 
had a good reason for telling them not to do something, that their teachers cared 
about them and what they think, and that their teachers did not get mad when they 
made mistakes. They were also likely to report that their teachers always tried to be 
fair, made them feel safe and comfortable, and could be trusted. Some students 
agreed while others disagreed that their teachers did not punish them without them 
knowing what happened and that their teachers kept their promises. Students 
assessed teacher-student trust in much the same way in 2001 (see Figure 42)., Slight 
year-to-year differences were not statistically different. Overall, there were no 
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differences between Annenberg and demographically similar non-Annenberg schools 
on this measure in any year. 



Figure 39. Teacher-Principal Trust: Average Standardized Change Unit Differences from 1994 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1994 to 2001 
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Figure 40. Teacher-Teacher Trust: Average Standardized Change Unit Differences from 1994 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1994 to 2001 
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Figure 4L Teacher-Parent Trust: Average Standardized Change Unit Differences 
from 1994 School System Mean for Annenberg and Demographically Similar Non- 
Annenberg Schools, 1994 to 2001 
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Figure 42. Student-Teacher Trust: Average Standardized Change Unit Differences from 1997 School 
System Average for Annenberg and Demographically Similar Non-Annenberg Schools, 1997 to 2001 
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Examples from the Field 

Relational trust among teachers, students, and parents grew stronger in several field 
research schools. In other schools, teachers developed more trusting relationships 
with each other and with their principals. In several, teachers became more 
comfortable working with staff members charged with implementing new curricula 
and teaching strategies. At the same time, there were several instances where staff 
turnover or behavior that betrayed expectations compromised growing trust 
relationships. 

Oscar Arias Sanchez Elementary School is an example of how difficult it can be 
to develop and sustain trust relationships. At the beginning of the field research, 
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teachers at Sanchez were not very comfortable inviting each other into their 
classrooms, nor were they comfortable discussing their teaching. Through the efforts 
of an in-house literacy coordinator, trust relationships among teachers began to 
develop. As these relationships grew, teachers became more willing to collaborate on 
the school’s literacy initiative and join in professional development activity. 

The literacy coordinator sought to build her relationships with teachers slowly. 

At first, she provided only the assistance that teachers requested. This way, she laid a 
foundation of trust for the work that followed. The coordinator viewed herself as a 
resource and a service provider. Because of her dependability, patience, and support, 
greater numbers of teachers began to seek her assistance and share their problems 
with her. Through their interactions with the coordinator, teachers began to open up 
and speak more frequently with one another about their classroom teaching. 

Over time, several factors began to undermine the trust the coordinator had 
begun to establish among teachers. Due to cuts in funding that supported her 
position, she had to reduce the amount of time she spent working with teachers. 
Without the coordinator’s regular involvement in their day-to-day work, some 
teachers began to withdraw from collegial activity back into their classrooms. At the 
same time, growing tensions between teachers and the principal began to undermine 
the coordinator’s work. Some teachers questioned whether the coordinator was 
working for their interests or for the principal’s. Despite these problems, the 
coordinator continued to work with a small group of teachers at the school and to 
deepen relationships among them. At the end of the field research, she remained 
optimistic that her progress would continue and that the trust relationships, while 
suffering some setbacks, would also continue to grow. 



Instructional Program Coherence 

School instructional program coherence is the second of the Model’s overarching 
Supports. It is defined by interrelated programs for students and staff that are guided 
by a common framework and pursued over a sustained period . 38 Strong program 
coherence is present when this common framework directs all aspects of student 
learning and governs the working environment of the school. Curriculum, 
instructional strategies, and student assessments are coordinated among grade-level 
teachers and across the school, showing a progression of more complex aspects of 
subject matter and intellectual challenge from one grade to the next. Key student 
support services such as tutoring, remedial instruction, parent education, and 
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opportunities for parent involvement are aligned with the framework and 
administrators and teachers hold each other accountable for its implementation. The 
school makes the framework the focus of its professional development efforts and 
allocates resources to its continued development. 

Schools with weak instructional program coherence lack a common framework. 
Their programs are fragmented and pull faculty and staff in different directions. 

There is little coordination among teachers within and across grade levels and student 
support programs do not necessarily promote the school’s instructional efforts. 

Faculty recruitment, hiring, accountability systems, and professional development are 
disconnected from any particular instructional focus. Different improvement 
initiatives may each address discrete problems, but there is little coordination among 
them to move the whole school forward. 



Development across Annenberg Schools 

Instructional program coherence in Annenberg schools was “moderate” in 1994 (see 
Table 10). Teachers were likely to agree but not strongly agree that they could see 
continuity from one program to the next at their schools, that their schools followed- 
up on the new programs they started, that curriculum and instruction were well 
coordinated across grades, and that curriculum and instruction were consistent 
among teachers at the same grade level. They were likely to agree that special 
programs do not just come and go and that their schools do not have too many 
programs to keep track of. They were also likely to agree that the coordination and 
focus of instruction had changed for the better in the past two years. In 2001, 
however, instructional program coherence was lower than in 1994, falling to the 
lower end of the measure’s “moderate” category. This decline occurred between 1999 
and 2001 (see Figure 43). Levels of coherence were much the same in Annenberg 
schools and in demographically similar non-Annenberg schools between 1994 and 
1999. In 2001, instructional program coherence in Annenberg schools had declined 
to a point significantly lower than coherence in non-Annenberg schools. 
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Table 10. Development of Instructional Program Coherence in Chicago Annenberg Schools, 
1 994 to 200 1 : Summary of Findings 





1994 


2001 


DIFFERENCE IN 
MEANS 


STANDARDIZED 
CHANGE UNIT 
DIFFERENCE 


NON-ANNENBERG 

COMPARISON 


Weakened 


Instructional 

Program 

Coherence 


Moderate 


Moderate 


-.0.27 


- 0.40 





Note: Measures are considered improved or weakened if the difference in means between 1994 or 1997 and 2001 is statistically 
significant at or beyond 0.01 (p < 0.01). Comparisons to non-Annenberg schools are for 2001. A “+” indicates that Annenberg 
schools were stronger than non-Annenberg schools on a particular measure (p < 0.01). A “a” indicates that the two groups of 
schools were statistically equivalent. A ” indicates that Annenberg schools were weaker than non-Annenberg schools on the 
measure (p < 0.0 1 ). 



Figure 43. Instructional Program Coherence: Average Standardized Change Unit Differences from 
1994 School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 



1994 to 2001 
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Examples from the Field 

Several field research schools worked specifically to increase the coherence of their 
instructional programs. Principals at these schools reduced the number of programs 
in their buildings, cutting ones that did not align well with the school’s mission and 
goals for development. In other schools, principals worked with teachers to 
coordinate curriculum and instruction within and across grade levels and promote 
greater commonality in teachers’ approach to instruction. These principals also 
supplied common curricular and instructional materials. Other schools did little to 
increase coherence; in fact, some increased program fragmentation by introducing 
new programs that had little to do with one another or with a central orienting focus. 
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Linus Carol Pauling School is an example of a school that strengthened its 
program coherence. The principal at Pauling worked actively to focus her teachers’ 
attention on a common curricular and instructional framework. She promoted this 
framework by finding appropriate resources, involving teachers in decisions 
concerning the framework, and giving teachers some measure of instructional 
autonomy within its parameters. 

In 1997 and 1998, Pauling housed many different academic programs and 
worked with an array of outside organizations. Although teachers thought many of 
these programs worked well, they also felt their number was overwhelming. By 1999, 
the principal had reduced Pauling’s initiatives to the one promoted by the school’s 
Annenberg External Partner, whose instructional philosophy matched the principal’s 
and who had worked with a group of teachers at the school for several years prior to 
the Challenge. Between 1998 and 1999, Pauling’s school improvement plan was 
revised substantially. Instead of cataloging many unrelated programs and activities, it 
promoted a single set of instructional practices. In 1999, the principal established a 
leadership team that involved more teachers in the decision-making process and, as a 
result, there was even greater commitment to the school’s instructional approach. 
Between 1999 and 2001, program coherence continued to strengthen as the 
principal and teachers at Pauling became increasingly committed to this instructional 
approach. Concurrently, teachers began to discuss their teaching practices in a shared 
language and in increasingly sophisticated terms and to explore integrating new 
methods into their instructional repertoires. 

In contrast, Andrei Sakharov Elementary School did little to achieve greater 
program coherence. From 1997 through 2001, Sakharov provided its teachers and 
students opportunities to participate in a variety of academic programs. From a 
university-supported mathematics curriculum, to at least three different reading 
initiatives, to arts projects, to several corporate-sponsored programs, to numerous 
opportunities for teachers to attend workshops and conferences, there was always 
something going on at the school. The principal was extremely entrepreneurial and 
was very successful at bringing in new funds and programs. 

Although all the programs at Sakharov had potential for improving instruction 
and student learning, there were simply too many programs and too little 
coordination among them. Neither Sakharov’s principal nor its Annenberg External 
Partner saw the many different programs as particularly problematic. Instead, they 
viewed them as offering opportunities to expose students to as much as possible and 
to offer teachers professional development and leadership opportunities. Regardless, 
teachers expressed frustration that they could not keep track of all the programs and 
that they lacked the time and effort to make any one work particularly well. One 
teacher observed, “There’s a lot going on in this school, but in little vacuums.” At 
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times, some of the programs conflicted with others in what they sought to 
accomplish. Teachers wanted some coherence and focus. In 1998, one teacher 
explained that having an overarching vision for the school would make it easier to 
organize the faculty’s work in a common direction and bring in and orient new 
teachers. Moreover, Sakharov’s school improvement plan was not used to guide 
decisions about which new programs the school would adopt. The LSC and teachers 
noted that the principal felt free to bring new initiatives into the school without their 
consultation. They observed that she often did so with great enthusiasm but with too 
little information and planning to implement them properly. Teachers described the 
principal s style as u She says ‘yes’ to everything” and “She just shoots from the hip.” 



Teacher Professional Development and Support for Change 

In addition to these seven Essential Supports, we examined changes in teacher 
participation in professional development, the quality of the professional 
development they experienced, and the support they felt they received from their 
principals and colleagues for change in their schools. In other Consortium research, 
these measures are sometimes considered part of teacher professional community and 
school leadership. This report discusses these measures separately because they 
represent important change mechanisms that may promote improvement in a 
number of other Supports. 

Teacher participation in professional development refers to the frequency with 
which teachers report that they participated in formal professional development 
activities during the course of a year. These include activities organized by teachers’ 
own schools, networks of teachers from other schools, outside professional groups or 
organizations, college and university courses, CPS workshops, and activities 
sponsored by the Chicago Teachers Union; Quality of professional development is 
the extent to which professional development addresses students’ needs; is sustained 
and coherently focused rather than short-term and unrelated; provides enough time 
to think carefully about, try, and evaluate new ideas; includes follow-up activities; is 
closely connected to schools’ improvement plans; and provides teachers with 
opportunities to work with peers in their own and other schools. 

Finally, support for change refers to the extent to which teachers believe that 
their principals and colleagues encourage them to take risks and try new instructional 
approaches. It also refers to the extent to which teachers perceive their schools as 
places where the faculty as a whole embraces improvement. The frequency with 
which teachers participate in high-quality professional development relates positively 
to a school’s orientation toward improvement, teachers’ classroom practices, the 
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implementation of change, and student academic achievement. 39 In addition, the 
literature indicates that the extent to which change is supported relates to risk-taking, 
experimentation, and improvement at the school and classroom levels. 40 

Overall, teacher participation in professional development activity was greater in 
2001 than in 1994 (see Table 1 1). The quality of professional development 
experienced by teachers in Annenberg schools also improved. At the same time, 
support for change in Annenberg schools declined. 



Table 11. Teacher Professional Development and Support for Change in Chicago Annenberg Schools, 
1 994 or 1 997 to 200 1 : Summary of Findings 





1994 OR 
1997 


2001 


DIFFERENCE 
IN MEANS 


STANDARDIZED 
CHANGE UNIT 
DIFFERENCE 


NON- 

ANNENBERG 

COMPARISON 


Improved 


Teacher 
Participation in 
Professional 
Development 


High 


High 


+ 0.15 


+ 0.47 




Quality of 

Professional 

Development 


High 


High 


+ 0.11 


+ 0.26 




Weakened 


Support for 
Change 


Moderate 


Moderate 


-0.34 


- 0.41 


s 



Note: Measures are considered improved or weakened if the difference in means between 1994 or 1997 and 2001 is statistically 
significant at or beyond 0.01 (p < 0.01). Comparisons to non-Annenberg schools are for 2001 . A “+” indicates that Annenberg 
schools were stronger than non-Annenberg schools on a particular measure {p < 0.01). A “a” indicates that the two groups of 
schools were statistically equivalent. A indicates that Annenberg schools were weaker than non-Annenberg schools on the 
measure (p < 0.01). 



Teacher Participation in Professional Development. In Annenberg schools, 
teacher participation increased between 1994 and 2001 with most of that increase 
occurring between 1994 and 1999 (see Figure 44). In 2001, more than half the 
teachers in Annenberg schools reported attending professional development activities 
at their school or at CPS-sponsored forums. More than half reported participating in 
networks outside of their school and discussing curriculum and instruction with an 
outside group. Between 20 and 50 percent attended union-sponsored activities or 
took university or college courses. In demographically similar non-Annenberg 
schools, teacher participation in professional development declined slightly between 
1994 and 1997 and then began to rise through 2001. In 1997 and 1999, it was 



39 Bryk et al. (forthcoming); Cohen and Hill (2000); Garet, Porter, Desimone, Birman, and Yoon 
(2002); Smylie et al. (2001); Sparks (1986); and Wiley and Yoon (1995). 

40 Fullan (2001); Hallinger and Heck (1996); Smylie, Conley, and Marks (2002). 
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greater in Annenberg schools than in non-Annenberg schools. By 2001, however, 
there was no statistical difference between the two groups. 

Quality of Professional Development. A similar pattern is present in the quality of 
professional development. In 1997, the average quality of professional development 
experienced by teachers in Annenberg schools was “high.” This means that teachers 
were likely to agree but not strongly agree that their professional development was 
closely connected to their schools’ improvement plans. They agreed that professional 
development provided them with opportunities to work with their colleagues and 
helped them understand their subject matter better. Their experiences were sustained 
and focused and included enough time to think about and judge new ideas. They 
found that their experiences addressed students’ needs. Some agreed and others 
disagreed that their professional development gave them opportunities to work with 
teachers at other schools. The quality of professional development experienced by 
Annenberg teachers increased between 1997 and 2001, especially between 1997 and 
1999 (see Figure 45). Nonetheless, it remained in the “high” category of the measure. 
Although it increased between 1997 and 1999 at a greater rate in Annenberg schools 
than in demographically similar non-Annenberg schools, this difference disappeared 
by 2001. None of the differences between the two groups were statistically significant 
in any year although the difference in 1999 approached significance. 



Figure 44. Teacher Participation in Professional Development: Average Standardized Change Unit 
Differences from 1994 School System Mean for Annenberg and Demographically Similar Non- 
Annenberg Schools, 1994 to 2001 
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Support for Change. In 1994, support for change in Annenberg schools was 
“moderate.” Teachers reported that they agreed or strongly agreed that their 
principals were willing to let them make changes, encouraged them to try new 
methods, and provided strong support for the changes that were introduced. They 
agreed that the principal encouraged them to take risks and pursue adequate 
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professional development to support the changes they were making. In addition, 
teachers agreed that the changes were supported by and involved many teachers. 
Levels of this measure in Annenberg schools declined between 1997 and 2001 but 
remained within the “moderate” category (see Figure 46). In 1999, support for 
change had become significantly stronger in Annenberg schools than in 
demographically similar non-Annenberg schools. By 2001, however, it declined to a 
point where it was no different from non-Annenberg schools. 



Figure 45. Quality of Professional Development: Average Standardized Change Unit Differences from 
1997 School System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 
1997 to 2001 
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Figure 46. Support for Change: Average Standardized Change Unit Differences from 1997 School 
System Mean for Annenberg and Demographically Similar Non-Annenberg Schools, 1997 to 2001 
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Summary 

In general, our findings on school development in Annenberg schools are mixed (see 
Table 12). As a group, schools participating in the Chicago Challenge were stronger 
on several measures of the Essential Supports in 2001 than they were in 1994 or 
1 997. At the same time, however, they failed to improve or grew weaker on other 
measures. The findings reveal no clear patterns of change among particular Essential 
Supports. That is, there were no Supports in which a predominant number of 
indicators were stronger or weaker in 2001 than in 1994 or 1997. 

The findings also indicate that there were virtually no statistically significant 
differences in the development of the Essential Supports between Annenberg and 
demographically similar non-Annenberg schools. Recall that in 1994, Annenberg and 
non-Annenberg schools were similar on every measure of the Essential Supports. In 
2001, there were only two measures for which there were statistically significant 
differences (see Table 13). Several measures of leadership and teacher professional 
community initially improved at a greater rate in Annenberg than demographically 
similar non-Annenberg schools. After 1999, however, these initial advantages 
disappeared and by 2001, there were no differences between the two groups on these 
measures. 
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Table 12. Summary of Changes in the Essential Supports in Annenberg Schools Between 1994 or 
1997 and 2001 





ESSENTIAL SUPPORT 


Improved 


Instruction 


Demand for Authentic Intellectual Work 
Writing Emphasis 
Interactive Instruction 


Learning Climate 


Classroom Personalism 
Safety 


Leadership 


Teacher Influence in Decision Making 


Teacher Professional Community 


Peer Collaboration 

Focus on Student Learning 

Orientation toward Innovation 


Parent-Community Involvement 


Teacher Outreach to Parents 
Parent Involvement in School 


Relational T rust 


Teacher-Principal Trust 
Teacher-Teacher Trust 
Teacher-Parent Trust 


Other 


Teacher Participation in Professional Development 
Quality of Professional Development 


No Net Change 


Instruction 


Didactic Instruction 


Learning Climate 


Press toward Academic Achievement 


Leadership 


Inclusive Leadership 

Principal Instructional Leadership 

Joint Problem Solving 


Teacher Professional Community 


Collective Responsibility 
Reflective Dialogue 


Parent-Community Involvement 


Teachers’ Use of Community Resources 
Teachers’ Ties to the Community 
Teachers’ Knowledge of Student Culture 
Human and Social Resources in the Community 


Relational Trust 


Teacher-Student Trust 


Weakened 


Learning Climate 


Peer Support for Academic Work 


Teacher Professional Community 


Teacher Commitment to School 


Other 


Instructional Program Coherence 
Principal-Teacher Support for Change 
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Table 13. Summary of Differences Between Annenberg and Demographically Similar Non-Annenberg 
Schools on Measures of the Essential Supports, 2001- 





ESSENTIAL SUPPORT 


Annenberg Schools Stronger 


Didactic Instruction 


Annenberg Schools Weaker 


Instructional Program Coherence 



The Case of Breakthrough Schools 

In 1999, the Chicago Challenge awarded new funding to 18 Breakthrough Schools 
from its 45 implementation networks with the expressed purpose of deepening 
development in those schools and helping them serve as models of development. 
Student outcomes and school development were examined among Breakthrough 
Schools between 1994 and 2001 and compared to student outcomes in other 
demographically similar Annenberg schools. These analyses controlled statistically for 
the same school characteristics and demographic variables as analyses for Annenberg 
schools as a whole. 41 



Student Outcomes 

As reported earlier in this section, there were no statistically significant differences 
between Annenberg and demographically similar non-Annenberg schools in student 
academic achievement or in student social and psychological outcomes. Similarly, 
there were virtually no statistically significant differences between Breakthrough and 
other Annenberg schools in these student outcomes. ITBS trends in Breakthrough 
Schools mirrored trends in other Annenberg schools (see Appendix G). So too did 
trends in student academic engagement, classroom behavior, social competence, and 
self-efficacy (see Appendix H). Only on the measure of student academic engagement 
was there a statistically significant difference in 2001 between Breakthrough and other 
Annenberg schools, and that difference favored non-Breakthrough schools. 



41 Due to the small number of Breakthrough schools, these findings include differences occurring at 
the 0.05 level of statistical significance as well as the 0.01 level of significance. 



101 



Part Two: Findings 



85 



School Development 

There was only one statistically significant difference between Breakthrough Schools 
and other Annenberg schools on any measure of the Essential Supports in 1999, the 
year Breakthrough Schools were identified. That difference was in teachers’ ties to the 
school community, a measure that was stronger for non-Breakthrough Annenberg 
schools. When compared in 2001, there were no significant differences between 
Breakthrough Schools and other Annenberg schools in instruction, student learning 
climate, parent and community involvement, or instructional program coherence. 
However, as a group, Breakthrough schools had become noticeably stronger than other 
Annenberg schools on most measures of teacher professional community and, to a 
lesser extent, stronger on measures of school leadership and relational trust (see Table 
14 and Appendix H). 



Table 14. Comparison of Breakthrough and Other Annenberg Schools on Measures of the 
Essential Supports, 2001 



BREAKTHROUGH SCHOOLS STRONGER 


Leadership 


Inclusive Leadership 

Joint Problem Solving 

Teacher Influence in Decision Making 


Teacher Professional Community 


Peer Collaboration 
Reflective Dialogue 
Focus on Student Learning 
Collective Responsibility 
Orientation toward Innovation 
Teacher Commitment to School 


Relational Trust 


Teacher-Principal Trust 
Teacher-Teacher Trust 


Other 


Quality of Teacher Professional Development 


NO DIFFERENCE 


Instruction 


All measures 


Student Learning Climate 


All measures 


Leadership 


Principal Instructional Leadership 


Parent Community Involvement 


All measures 


Relational Trust 


Teacher- Parent Trust 
Teacher-Student Trust 


Other 


Instructional Program Coherence 

Teacher Participation in Professional Development 

Support for Change 



Trends in measures of teacher professional community indicate that 
Breakthrough Schools were slightly stronger than other Annenberg schools on 
measures of professional community in 1999, but these differences were not 
statistically significant (see Figures 47 through 52). By 2001, however, peer 
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collaboration, focus on student learning, collective responsibility, reflective dialogue, 
orientation toward innovation, and teacher commitment to school were significantly 
stronger among these schools. These differences resulted from a continuous, albeit 
slight, upward trajectory among Breakthrough Schools and a downturn after 1999 
among other Annenberg schools. 



Figure 47. Peer Collaboration: Average Standardized Change Unit Differences from 1994 School 
System Mean for Breakthrough and Other Annenberg Schools, 1994 to 2001 
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Figure 48. Reflective Dialogue: Average Standardized Change Unit Differences from 1994 School 
System Mean for Breakthrough and Other Annenberg Schools, 1994 to 2001 



e v 
S' 3 

Q ^ 

i 

£ ^ 
STcn 

s 



* 1 Breakthrough Schools 
** C** "Other Annenberg Schools 



1994 



1997 



1999 



2001 




103 



Part Two: Findings • 87 



Figure 49. Focus on Student Learning: Average Standardized Change Unit Differences from 1994 
School System Mean for Breakthrough and Other Annenberg Schools, 1994 to 2001 




Figure 50. Collective Responsibility: Average Standardized Change Unit Differences from 1994 
School System Mean for Breakthrough and Other Annenberg Schools, 1994 to 2001 




Figure 51. Orientation toward Innovation. Average Standardized Change Unit Differences from 1994 
School System Mean for Breakthrough and Other Annenberg Schools, 1994 to 2001 
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Figure 52. Teacher Commitment to School: Average Standardized Change Unit Differences from 
1994 School System Mean for Breakthrough and Other Annenberg Schools, 1994 to 2001 
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While not as pronounced as those for teacher professional community, similar 
differences were also found between Breakthrough and other Annenberg schools on 
several measures of school leadership (see Figures 53 through 55). In the baseline 
years and 1999, Breakthrough Schools were slightly stronger on measures of the 
inclusiveness of school leadership, joint problem solving, and teacher influence in 
decision making, though these differences were not statistically significant. In 2001, 
however, Breakthrough Schools were stronger on these measures. Like teacher 
professional community, differences in school leadership, with the exception of teacher 
influence in decision making, resulted from gradual improvement among 
Breakthrough Schools occurring at the same time that levels among other Annenberg 
schools were declining. For teacher influence, the 2001 difference occurred because the 
decline in this measure among Breakthrough Schools was not as steep as the decline 
among other Annenberg schools. 



Figure 53. Inclusive Leadership: Average Standardized Change Unit Differences from 1994 School 
System Mean for Breakthrough and Other Annenberg Schools, 1994 to 2001 
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Figure 54. Joint Problem Solving: Average Standardized Change Unit Differences from 1997 School 
System Mean for Breakthrough and Other Annenberg Schools, 1997 to 2001 
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Figure 55 . Teacher Influence in Decision Making: Average Standardized Change Unit Differences 
from 1994 School System Mean for Breakthrough and Other Annenberg Schools, 1994 to 2001 
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Finally, Breakthrough Schools surpassed other Annenberg schools in 
development of teacher-principal trust and teacher-teacher trust (see Figures 56 and 
57). Breakthrough Schools were slightly stronger on these measures than other 
Annenberg schools between 1994 and 1999, but these differences were not 
statistically significant. After 1999, Breakthrough Schools were able to sustain and 
build upon their initial levels of trust while levels of trust declined in other 
Annenberg schools. The same pattern was found with respect to the quality of 
teacher professional development, (see Figure 58). 

The relative success of Breakthrough Schools is discussed in Part Three of this 
report. 
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Figure 56. Teacher- Principal Trust: Average Standardized Change Unit Differences from 1994 School 
System Mean for Breakthrough and Other Annenberg Schools, 1994 to 2001 
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Figure 57. Teacher-Teacher Trust: Average Standardized Change Unit Differences from 1994 School 
System Mean for Breakthrough and Other Annenberg Schools, 1994 to 2001 
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Figure 58. Quality of Teacher Professional Development: Average Standardized Change Unit 
Differences from 1997 School System Mean for Breakthrough and Other Annenberg Schools, 
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A Closer Look at What Makes 
School Improvement Successful 

As described in Part One of this report, this study of the Chicago Annenberg 
Challenge included both macro and micro levels of inquiry. Comparative religion 
scholar Diana Eck provides another way to think about the design of our work in her 
discussion of how people view icons . 42 According to Eck, icons can be seen as 
“objects,” entities in and of themselves that can be described and assessed. On the 
other hand, icons may also be considered “windows” through which people gain new 
insight and understanding. 

The findings reported thus far consider the Chicago Challenge as an “object,” a 
large-scale reform initiative to promote local school development whose work across 
a substantial number of schools can be documented and assessed. Analyses of 
citywide survey and standardized test-score data described and assessed trends in 
student outcome measures and indicators of school development among Annenberg 
schools, and those trends were compared to demographically similar non-Annenberg 
schools. Analyses of survey and test-score data also compared Breakthrough Schools 
to other Annenberg schools. Data from field research schools illustrated aspects of 
school development found in the analyses. 

At the same time, the Challenge is also a “window” through which individual 
schools can be studied to better understand how school development may be 



42 Eck (1993). 
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promoted or constrained. The field research was designed with this purpose in mind. 
This last section of findings presents a view of local school development through the 
“Annenberg window.” This is not a view of school development as a function of a 
large-scale reform initiative; instead, it is a view of development from the perspective 
of individual schools and the work they do to get better. Data from longitudinal field 
research in 12 Annenberg elementary schools were analyzed to reveal emergent 
themes and patterns of activities and conditions that were associated with both 
successful and stagnant development. (See Part One and Appendix C for more 
information on the field research methodology.) 

Unlike the previous section in which field research schools were referred to by 
pseudonyms, we refer to schools in this section by letter and group. We use letters 
and groups here in order to make more clear the presentation of cross-school 
findings. In addition, using letters and groups in this section serves to protect the 
confidentiality of our field research sites by minimizing the possibility that 
information from this section could be combined with information from the 
previous section to reveal school identities. 



Promoting School Development 

Four patterns of development were identified across the 12 field research schools 
during the five years of this study. These patterns are shown in Figure 56. This figure 
is a heuristic. The lines represent general directions of development; they do not 
indicate actual magnitudes of change nor relative differences in starting or ending 
points among the schools. The first pattern is illustrated by two schools — Group 
1 — that were relatively high on measures of the Essential Supports in 1997 and did 
not change in any appreciable way during the five-year study. Two other schools that 
developed continuously on one or more of the Essential Supports between 1997 and 
2001 — Group 2 — illustrate the second pattern. Four schools that developed on one 
or more of the Essential Supports between 1997 and 1999 but then regressed 
between 1999 and 2001 — Group 3 — illustrate the third pattern. Four more schools 
that were quite low on indicators of the Essential Supports in 1997 and failed to 
develop in any appreciable way — Group 4 — illustrate the last pattern. 
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Figure 59. Patterns of Development among Field Research Schools 



High 



Group } 




Low » Group 4 



The criterion for classifying schools into these groups was their development on 
one or more of the Essential Supports. We also identified general patterns of student 
achievement in terms of average changes in the percentages of students scoring at or 
above national norms in reading and mathematics on the ITBS between 1997 and 
2001 . This is not a particularly strong measure of achievement and our analysis did 
not control for school characteristics and demographic factors like the other 
achievement analyses in this report. The purpose here is simply to illustrate general 
relationships between different patterns of school development and student academic 
achievement as suggested by the logic of the Model of Essential Supports. 

Following this logic, it would be expected that achievement in schools in Groups 
1, 2, and 3 would be greater than achievement in schools in Group 4. It would also 
be expected that Group 2 schools, those that made continuous progress in 
development of the Essential Supports, would make the greatest improvement in 
achievement. As shown in Table 13, average improvement in Groups 1, 2, and 3 was 
greater than in Group 4. And as expected, Group 2 showed greater average 
improvement than Groups 1 and 3. 

While these general relationships are what might be expected, they do not 
illustrate very well how achievement and school development are related. To examine 
these relationships in more detail, achievement in Group 2 was compared to 
achievement in Group 3 during two periods. The first was from 1997 to 1999, the 
second from 1999 to 2001 . Group 3 improved on the Essential Supports during the 
first period and regressed during the second. Group 2 developed continuously during 
both. As shown in Table 16, improvement in average achievement in Group 2 was 
somewhat greater in the second period than in the first. On average, Group 3 
improved academic achievement as it was developing the Essential Supports and 
average achievement declined slightly as those schools regressed in their development. 
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Table 15. Average Change in the Percentage of Students Scoring At or Above National Norms on the 
ITBS in Reading and Mathematics by School Development Group, 1997 to 2001 





READING 


MATH 


Group 1 


+ 10.1 


+ 7.3 


Group 2 


+ 18.0 


+ 13.6 


Group 3 


+ 8.5 


+ 10.0 


Group 4 


+ 5.8 


+ 1.5 



Table 16 . Average Change in the Percentage of Students Scoring At or Above National Norms on the 
ITBS in Reading and Mathematics in Groups 2 and 3, 1997 to 1999 and 1999 to 2001. 





1997 TO 1999 


1999 TO 2001 


Group 2 


Reading 


+ 8.55 


+ 9.45 


Math 


+ 5.90 


+ 7.70 


Group 3 


Reading 


+ 11.32 


-2.88 


Math 


+ 11.43 


- 1.40 



When examining the field research about schools’ efforts to develop, four general 
findings emerged. First, higher levels of school development and continuous 
improvement were associated with coordinated or concerted attention to multiple 
Essential Supports. Second, higher levels of development and continuous 
improvement were associated with the use of multiple, reinforcing strategies for 
change. Third, higher levels of development and continuous improvement were 
associated with a strong base of external resources aligned with the school’s 
development agenda. Finally, higher levels of development and continuous 
improvement were associated with strong, broad-based, and distributed leadership. 
Whether these findings were true for a school was, with few exceptions, associated 
with its group classification and pattern of development. In general, these findings 
were consistently true or more true than false of schools that were more highly 
developed on the Essential Supports or developed on the Supports over the course of 
the study. They were consistently not true of schools that were relatively weak in the 
Supports in 1997 and failed to develop. And they were true in 1999 but only 
partially true (i.e., only one or two were true) in 2001 for schools that developed 
between 1997 and 1999 and regressed thereafter. This suggests that the different 
aspects of a school development process work in conjunction and that if one or two 
fall away, development may be compromised. 

These patterns of findings are summarized in Table 17. The columns in this table 
show the four groups of schools according to their patterns of development. The 
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rows are the major findings that distinguish groups of schools from each other. The 
cells within the table show the schools in each group and indicate with “+’s” and “-‘s” 
whether a particular finding was true or more true than false about the school or 
whether the finding was false or more false than true about the school in a particular 
year. The table shows a combination of “+Y* and “-‘s” for each school in each cell. 
The first “+” or indicates whether the finding was true or (more true than false) 
or false (or more false than true) for that school in 1999. The second “+” or 
indicates whether the finding was true or false for that school in 2001. For example, 
for School A (Group 1), the first finding was true for this school in both 1999 and 
2001. For School E (Group 3) , the first finding was true for this school in 1999 but 
false in 2001. 



Table 17. Relationships Between Patterns of School Development and Findings on School 
Development Processes 



FINDINGS 


GROUP 1 
HIGHER INITIAL 
STATE, NO 
DEVELOPMENT 


GROUP 2 
ONGOING 
DEVELOPMENT 


GROUP 3 
INITIAL 

DEVELOPMENT 
THEN REGRESS 


GROUP 4 
LOW INITIAL 
STATE, NO 
DEVELOPMENT 


School targets multiple 
Essential Supports in a 
concerted or 
coordinated manner. 


School A + + 
School B - + 


School C + + 
School D + + 


School E + + 
School F + + 
School G + - 
School Fi + - 


School I - - 
School J - - 
School K - - 
School L - - 


School uses multiple, 
complementary change 
strategies . 


School A + + 
School B + + 


School C + + 
School D - + 


School E + + 
School F + - 
School G + - 
School H + - 


School I - - 
School J - - 
School K - - 
School L + - 


School has strong, 
aligned base of 
external resources. 


School A + + 
School B - + 


School C + + 
School D + + 


School E + - 
School F + - 
School G + - 
School H + - 


School I - - 
School J - - 
School K - - 
School L + - 


School develops strong 
distributive leadership. 


School A + + 
School B + + 


School C + + 
School D + + 


School E + - 
School F + - 
School G + - 
School H + - 


School I - - 
School J - - 
School K - - 
School L + - 



Targeting Multiple Essential Supports 

The first finding from this analysis indicates that school development is associated 
with a coordinated focus on multiple Essential Supports. In the most highly 
developed schools and in schools that developed continuously (Groups 1 and 2), 
change initiatives focused on development of several related Supports which created 
synergy to promote or sustain overall school development. When schools focused on 
a single Support, or when they focused on multiple Supports in an uncoordinated 
manner, little overall development occurred (Group 4). When schools shifted their 




112 



96 



The Chicago annenberc Challenge: Successes, Failures, and Lessons for the future 



focus from multiple Supports to only one, or when efforts lost momentum or 
coordination, initial improvement declined (Group 3). 

Why would a school be more likely to develop by targeting multiple supports? As 
discussed earlier, and as other Consortium research suggests, the Essential Supports 
are not discrete, independent elements. 43 Rather, they operate as related parts of a 
system. The Supports that represent key organizational capacities — school leadership, 
professional community, and parent and community support — are crucial for 
developing and supporting school practices — student learning climate and quality 
instruction — that in turn are instrumental for promoting student learning. This logic 
is consistent with the discussion in the next section about the Challenge’s overall 
reform strategies and with other literature showing that school development requires 
long, steady work not focused solely on the implementation of specific programs and 
policies, but on the broader, coherent development of school organization and 

• 4 A 

practices. 

Field research documenting the first three years of the Chicago Challenge 
suggested that the success of efforts to develop learning climate and instruction is 
contingent on previous or concurrent development of school organizational 
capacity. 45 For example, strong leadership is necessary to create and sustain a well- 
paced, challenging, and coherent instructional program. 46 There must be a strong 
professional community of teachers who work together to coordinate the curriculum, 
achieve consistency in expectations for student learning, develop intellectually 
rigorous tasks, and engage students in those tasks. It is unlikely that such a 
professional community can thrive over time if school leadership does not help 
develop it and provide enough time and resources to get its work done. Overall then, 
focusing on one Essential Support may promote development of that particular 
support, but development is likely to be limited and difficult to sustain if there are 
weaknesses in others. 

Two field research schools illustrate these points. School J’s failure to develop 
other Essential Supports undermined its efforts to develop its instructional program. 
When this school began working with its Annenberg partner in 1997, it focused on 
raising the quality of reading instruction. Even though initial efforts were promising, 
weak school leadership and teacher professional community soon compromised 
them. School H made concerted efforts to develop multiple Essential Supports, albeit 
without much coordination or a clear overarching vision. Its Annenberg External 



43 Bryk et al. (forthcoming). 

44 For example Elmore and McLaughlin (1988); Fullan (2001); and Louis and Miles (1990). 

45 Wenzel et al. (2001). 

46 See Newmann et al. (2001b). 
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Partner organized a group of teachers to develop professional community, promote 
professional development, and improve student learning climate and instruction. At 
the same time, the principal focused his energy on increasing student test scores, 
improving student discipline and safety, and promoting small group instruction. 
Although the Annenberg teachers and the principal focused on different areas of 
school development, their work converged around the promotion of small group 
instruction. These efforts helped School H develop between 1997 and 1999 but 
progress began to disintegrate between 1999 and 2001 as development efforts began 
to diverge and conflict. Teachers working to promote more small group instruction 
faced a principal who began to assert student discipline as the school's first priority. 
The principal quashed teacher efforts to develop smaller, more flexible instructional 
groups because those efforts required that students move among self-contained 
classrooms. The principal believed that such movement provided too great a chance 
for disruption and student misconduct. 

School H's regress was also related to a shift toward reliance on one change 
mechanism. Initially working to achieve change through teacher professional 
development and student test score accountability, test-score accountability began to 
take precedence. In addition when its External Partner began to withdraw from the 
school it lost a key resource for its development efforts. Finally, where teachers once 
shared in leadership for school development, the principal began to consolidate his 
control over school decisions. 

Employing Multiple, Reinforcing Strategies 

The second finding is consistent with the literature on educational change that 
concludes that there are no “quick fixes” or “cookbook solutions” for school 
development. 47 Like the literature, our field research indicates that successful school 
development is achieved not just from the “top down” or “bottom up,” but also from 
the “inside out” through a combination of strategies that most effectively develop 
teachers' “will” and “skill.” 48 There was no single program or initiative that provided 
any of the field research schools with everything they needed to develop; instead of 
reliance on a single solution, school development was associated with employing 
idiosyncratic combinations of complementary, mutually reinforcing strategies. 

As will be discussed in greater detail in the next section of this report, literature 
on education reform identifies three types of mechanisms that may promote change 
at the school and classroom levels. 49 The first consists of bureaucratic and normative 



47 For example, Fullan (2001) and Maeher and Midgley (1996). 

48 Newmann and Wehlage (1995); Sarason (1990); and Tyack and Cuban (1995). 

49 Hannaway (1993) and Smylie and Perry (1998). 
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controls and sanctions that seek to compel individuals and schools to take specific 
actions. The second consists of incentives to prompt voluntary action. The third 
consists of learning opportunities that develop new knowledge and skills and, from 
that development, evoke new action. 

Across the field research schools, there were many examples of these mechanisms 
in effect. Some principals and External Partners offered teachers incentives to adopt 
and develop commitment to new teaching practices. These came in the form of 
monetary stipends, public praise and encouragement, time to work with colleagues or 
pursue professional development, consultations with experts, increased classroom 
autonomy, and opportunities to exercise greater influence in decision making. 
Numerous opportunities for learning and development were available to teachers, 
principals, and other school staff in our study schools. These included workshops and 
conferences, collaborative planning and work groups, networking with teachers from 
other schools, working with in-house curriculum coordinators, new mentoring 
relationships, access to professional journals, and increased opportunities for collegial 
interaction. There were also a number of controls at work. In most of the field research 
schools, CPS student retention and school probation policies were highly influential 
sources of accountability and control for both principals and teachers. A number of 
principals created additional monitoring and accountability systems. Several 
developed and enforced their own set of expectations for staff and student 
performance. At one school, the External Partner instituted a formal review process 
that made staff members publicly accountable to the Partner and each other. In 
several others, the growth of teamwork and collaboration, along with the expansion 
of teachers’ leadership, reinforced collegial accountability and control. 

Both the literature and the project’s field research indicate that no mechanism 
alone is likely to promote and sustain school development over an extended period of 
time. 50 In the field research sample, continuously developing schools and schools that 
were initially strong and steady in their development (Groups 1 and 2) were more 
likely than nondeveloping schools (Group 4) to use a variety of strategies to trigger 
development, but they did not use them in any common combination or order. 
Different mechanisms were instrumental in sparking development activity in each of 
the schools. Some were motivated to act by the threat of administrative sanction; 
others were prompted by the adoption of a promising new approach to teaching. In 
no instance were the mechanisms that initiated activity adequate to sustain 
development over an extended period of time without the introduction of others. For 
example, School D accelerated its development between 1999 and 2001 in large part 
because it introduced a broader range of change mechanisms. On the other hand, 



50 See Smylie and Perry (1998). 



215 



Part Two: Findings 



99 



loss of progress among all but one school in Group 3 was associated with movement 
away from a coordinated combination of change mechanisms and increased reliance 
on one — bureaucratic accountability through high-stakes student testing. 

Although no patterns were detected in the specific strategies that developing 
schools used, it is likely that a school’s particular situation may call for specific 
combinations or for certain mechanisms to be used before others. For some, the most 
effective means to initiate change might be the introduction of a new accountability 
system or the replacement of the principal or members of the teaching staff. For 
others, this strategy could be completely ineffective. Likewise, professional 
development might motivate teachers at one school to adopt new practices, but be 
largely ignored at another. The apparent context-specific, idiosyncratic nature of 
effective strategies requires additional investigation. For now, it seems that evocation 
of effective combinations of strategies depends on understanding the strengths and 
weaknesses of a particular school and the needs and interests of the people who work 
there. 51 At the micro-level, these observations axe consistent with our earlier discussion 
about the alignment of reforms with schools’ capacity to implement them well. 



Securing External Resources 

As will be discussed in more detail in Part Three, school development requires many 
different types of resources. These include people, time, money, and materials. They 
also include ideas and expertise, leadership, political support, beliefs and values, and 
social trust. Which new external resources a school may need is dependent upon the 
areas it seeks to develop, the strength of its internal resources, and the external 
resources it has already accumulated. 

External resources for school development may come from a variety of 
places — the central administration, groups working with the school, community 
organizations, and parents. Underresourced and underdeveloped schools may depend 
a great deal on external resources to promote development. Indeed, as we argued 
earlier, failure to secure and sustain adequate external resources may thwart 
development efforts. 52 

Schools in the field research sample drew from several different sources of 
support. Although many worked with multiple outside organizations and other 
service providers, CPS and the Chicago Challenge stood out as the most 
predominant sources of external support. Beyond supporting basic school operations, 



51 Evans (1996); Hargreaves (2003). 

52 Fullan (2001). 
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CPS provided several of the field research schools with budget directors, instructional 
consultants, and probation managers. Moreover, the system’s capital improvement 
initiative funded badly needed repairs, renovations, and new construction at several 
schools. 

The Chicago Challenge linked schools with new human and intellectual 
resources and provided modest financial support for school development. External 
Partners brought ideas and expertise, focus, and impetus to promote school 
development. 53 Partners could also expand the intellectual and social resources that 
were available to schools by linking them with other schools engaged in similar 
development activity. Annenberg grants, while averaging little more than 1 percent 
of a school’s operating budget, were used to purchase important resources for school 
development such as in-house curriculum coordinators, teacher professional 
development, classroom libraries, and new instructional materials. The Challenge 
also provided some professional support in the form of workshops, conferences, and 
consultations with its staff. Finally, participation in the Challenge helped some 
schools lever additional resources. Such was the case among several schools that were 
working with their External Partners to increase parent involvement and cultivate 
stronger, more supportive relationships with organizations in their communities. 

The field research reveals a more complicated story about the relationship 
between securing additional resources and school development, however. 
Continuously developing and more highly developed schools (Groups 1 and 2) were 
generally more effective than nondeveloping schools (Group 4) at searching for, 
securing, and taking full advantage of external resources. At the same time, what 
distinguished Groups 1 and 2 from Group 4 was not simply entrepreneurial 
capability. Some nondeveloping schools were quite accomplished at obtaining 
external resources. Rather, it was the ability of Groups 1 and 2 to secure resources 
aligned with a particular development agenda and to employ those resources in an 
efficient and strategic manner that differentiated them from Group 4. Group 3 
illustrates these points well. In those schools the loss and fragmentation of key 
resources was associated with regress. 

Two of the nondeveloping schools in Group 4 had relatively few resources and it 
was apparent that this constrained their efforts to develop. On the other hand, two 
schools in that group had substantial resources, but these were acquired in an 
indiscriminate manner and were not coordinated with their schools’ development 
agendas. These schools did not always use their resources to their full potential. 
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Distributing Leadership for School Development 

The first three findings from the field research focused on aims, strategies, and 
resources for school development. The fourth focuses on the individuals who led 
development efforts. When the field study schools were examined closely, it was 
apparent that the strength and breadth of leadership distinguished more highly 
developed and developing schools (Groups 1 and 2) from nondeveloping ones 
(Group 4). Schools that made the greatest progress were those that cultivated strong, 
distributive leadership. Poorly developed and nondeveloping schools were likely to 
have a single source of consolidated leadership or simply have weak overall 
leadership. In schools that regressed after initial development (Group 3), leadership 
(usually the principal’s) that was once strong and distributed grew weaker or became 
more authoritarian and consolidated. These findings are consistent with other studies 
of distributive leadership and the implementation and institutionalization of complex 
educational change. 54 

In all of the field research schools, development was more likely to occur when 
key leadership tasks were performed in a coordinated manner by multiple actors in a 
school community, including the principal, teachers, outside organizations working 
with the school, coordinators, and parents. These tasks include: (a) creating and 
sustaining a vision for school development across multiple Essential Supports; (b) 
engaging others in school development initiatives; (c) promoting coherence among 
those initiatives; (d) providing incentives and opportunities to develop staff 
knowledge and skills; (e) developing curriculum and student assessments; (f) 
monitoring, providing encouragement, and holding staff members accountable for 
progress made toward school development; (g) obtaining external resources to 
support the school’s development agenda; and (h) managing external influences in 
ways that support development. Most of these tasks relate to the first three findings. 

While this analysis points to the importance of the distributive performance of 
such tasks to school development, it also highlights the “make-or-break” role that 
principals play in school development. 55 In the field research schools, principals were 
often at the heart of successful development activity. The most effective principals 
performed a number of common leadership tasks. They may not have performed 
them alone, but they performed them nonetheless. They articulated a clear, coherent 
vision of strong instructional practice and effective school organization. They 



53 Newman n and Sconzert (2000). 

54 For example, Heller and Firestone (1993); Mayrowetz and Weinstein (1999); and Spillane, 
Halverson, and Diamond (2001). See also Sebring, Hallman, and Smylie (2003) for further analysis of 
schools where distributed leadership was reconsolidated in the principal. 

55 The importance of the role of principals is echoed by Chicago Annenberg External Partners in the 
report Sconzert, Wenzel, and Smylie (2003). 
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communicated high expectations for teachers as both instructors and leaders of 
development, and they pressed teachers to meet those expectations. These principals 
persistently promoted the development of professional competence and leadership 
capacity among staff members and could be counted on to provide resources to 
support that development. Principals of developing schools distributed leadership 
among others and managed their “leadership work.” At the same time, they could be 
forceful and directive to ensure that the school stayed focused and that work was 
completed. 

Principals in more highly developed and continuously developing schools 
managed external resources effectively. They obtained the human, intellectual, and 
material resources needed to support development efforts. They established strong, 
productive relationships with their External Partners and with CPS administrative 
staff. These principals effectively protected their schools from external distractions 
and interference. And, when distraction and interference did intrude, they worked to 
minimize any disruptive effect. Principals were also among the first in the school 
community to feel the sparks of external pressure and opportunities for school 
development. Because they had the opportunity to marshal external support, 
principals could couple the initiation of development activity with new resources to 
fuel it. Finally, because of their position of authority within the school and between 
the school and its environment, principals could bring coherence among school 
development goals, strategies, and internal and external resources. 

The experiences of the field research schools also suggest that teacher leaders can 
be powerful change agents for school development when they work with their 
principals. In developing schools, teachers contributed expertise, skills, and 
perspectives on problems. They helped to create and sustain a vision for school 
development, and their assistance was crucial in promoting and engaging other 
teachers in development initiatives. Teacher leaders led professional development 
activities, monitored and held staff accountable for improving their practice, and 
helped the school obtain external resources. 

One particularly notable example of distributed leadership was the creation of 
full-time in-house coordinator positions that focused primarily on the development 
of classroom curriculum and instruction. Half of the field research schools had at 
least one in-house coordinator. These coordinators were usually teachers at the school 
who were released from their classroom duties to help their colleagues develop 
classroom practice. Selected because of their teaching ability and their ability to work 
well with others, the coordinators were usually trained by Annenberg External 
Partners to lead professional development activities and mentor teachers as they 
implemented the Partners’ curricular and instructional programs and practices at the 
classroom level. 
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The specific work the in-house coordinators performed varied, but the creation 
of these positions usually led to growth in overall school leadership. Coordinators 
held workshops, worked individually with teachers, observed classroom practice, and 
obtained new curricular and instructional materials. They became focal points for 
professional development. Teachers in some schools began to turn more often to 
them than to their principals for instructional expertise and assistance. Coordinators 
served as liaisons between teachers and principals and they facilitated communication 
between their schools, External Partners, and other schools in their Annenberg 
networks. Coordinators performed these roles particularly well in several of the more 
highly developed and continuously developing schools. Indeed, in several schools the 
loss of effective in-house coordinators was a primary cause for their regress. 



Implications 

These findings raise several important implications for promoting school 
development. First, principal leadership matters in promoting school development 
and it matters a lot. It is not simply any form of principal leadership that is effective; 
it should be inclusive, distributive, and visionary. Even though it is important that 
other members of a school community become involved in “leadership work,” the 
principal occupies a unique position in school organizations to initiate, manage, and 
sustain development. The principal is crucial in developing leadership capacity 
among staff and in distributing and managing the performance of leadership tasks by 
others. While strong principal leadership alone may not be sufficient to promote and 
sustain school development over time, it is clearly necessary. There is some debate in 
the literature about the importance of the principal in the context of distributed 
leadership for implementing and institutionalizing complex innovation . 56 In this 
study, however, there is no debate. Principals played a “make or break” role in 
promoting and achieving school development. 

Second, these findings point to the need for school leadership to think 
systemically about school organization and development. The most successful schools 
in the field research were those that targeted for development multiple, mutually- 
reinforcing aspects of school organization and practice. In order to set and pursue 
such an agenda, leadership must see school organizations in terms of their dynamic 
interdependent parts . 57 Leadership must understand how these parts work and 
change together and how they can support each other in promoting effective 
teaching and student learning. Leadership must understand the dynamic quality of 



56 For example, Heller and Firestone (1995) and Mayrowetz and Weinstein (1999). 

57 See Bolman and Deal (1997) and Bryk et al. (forthcoming). 
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school organizations and how change in one aspect can have positive or negative 
consequences in others. 

Third, these findings point to the importance of organizing development efforts 
around strong maps or theories of school development and change. In addition, it 
should evoke a complementary array of strategically chosen change mechanisms 
tailored to the school s particular organizational strengths and weaknesses, its 
development goals, and the needs and interests of its community. Strategies to 
develop individual and collective capacity — the “will” and the “skill” — for leadership, 
for organizational development, and for improved classroom practice appear vital to 
successful development efforts. The findings indicate, however, that strategies to 
develop capacity may not be sufficient to promote development over time without 
complementary incentives and systems of accountability that reinforce efforts to 
develop and enact new capacity. Likewise, the findings suggest that leadership would 
be ill advised to rely exclusively on accountability mechanisms to achieve much 
school development. Although development in some schools may need a “kick-start” 
from a high-stakes accountability system, it is unlikely that such an incentive will 
have long-lasting effects without the introduction of other strategies. 

Fourth, these findings point to the importance of coherence and to leadership's 
role in achieving it. Leadership, particularly principal leadership, is crucial in the 
alignment of development goals, strategies, and internal and external resources 
around a strong, robust vision of a good school, good teaching, a learned student, 
and a sound theory of change. As mentioned earlier, by virtue of the authority of 
their roles and their access to and control of resources, principals are in a unique 
position in the school organization to promote such coherence. The alternative is 
fragmentation that, according to our study and other research, can become a serious 
impediment to school development . 58 

Finally, as will be discussed further in Part Three, these findings indicate that 
school development takes time and requires long steady work. Progress is fragile and 
initial gains can be lost, sometimes easily. Beyond patience and persistence, 
sustainable school development requires a stable base of resources, ongoing 
monitoring and assessment of development goals, progress toward achieving those 
goals, and effective development strategies. Development may also require flexibility 
and adjustment of goals, strategies, and resources as conditions change inside and 
outside the school. Without a base of human and social resources — leadership, 



58 See FuIIan (2001) and Newmann et al. (2001b). 
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professional community, parent and community involvement, and relational 
trust — it is difficult to imagine that local school development efforts will get very far 
or last very long. 
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Part Three: Explaining the 
Findings 



The previous section examined how the Chicago Annenberg Challenge 
promoted improvement in Chicago public schools and the extent to 
which improvement occurred. This section examines those findings 
further and provides an assessment of the overall impact of the 
Challenge. It also presents a number of explanations for the successes and 
failures of the Challenge as a large-scale reform initiative. 



Review of Findings 

Part Two presented findings concerning the Chicago Challenge’s “bottom line”: 
improvement in student academic achievement and nonacademic outcomes. Our 
evidence suggests that among the schools it supported, the Challenge did not 
achieve this goal 

■ Analyses of ITBS scores reveal that between 1996 and 2001, student achievement 
improved overall across Annenberg schools. This was similar to improvement 
across the system. 

■ During the same period, rates of gain in student achievement among Annenberg 
schools did not improve markedly. Across grade levels, the size of one-year 
achievement gains remained constant or fluctuated slightly. In other words, at 
the end of the Challenge, students in Annenberg schools achieved at much the 
same rate as at the beginning. This pattern was evident in both reading and 
mathematics, although overall rates of gain in reading were slightly larger than 
rates of gain in math. There were no statistically significant differences in student 
achievement between Annenberg schools and demographically similar non- 
Annenberg schools. This indicates that that there was no Annenberg effect on 
achievement. 
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Analyses show both positive and negative changes across Annenberg schools among 
different social and psychological student outcomes. 

■ Initially, student academic engagement improved among Annenberg schools but 
then fell to a point where it was only slightly higher in 2001 than in 1994. 

■ Students’ sense of self-efficacy first weakened and then strengthened, but 
remained weaker in 2001 than in 1997. 

■ Both classroom behavior and social competence among students in Annenberg 
schools declined slightly between 1994 and 2001. 

■ Like student academic achievement, there were no statistically significant 
differences in these student outcomes between Annenberg and demographically 
similar non-Annenberg schools, which indicates that there was no Annenberg 
effect on these outcomes. 

Despite these findings, it nonetheless remained important to examine trends in 
school development. The Challenge’s logic and the logic inherent in the Model of 
Essential Supports suggest that before improvement in student outcomes can occur, 
schools need to develop in ways that promote it. Therefore, it was important to 
determine whether Annenberg schools developed in ways that would lay the 
foundation for subsequent improvement in student outcomes and whether 
development among Annenberg schools as a group was greater than development 
among schools that did not participate in the Challenge. 

The findings present a somewhat complicated story. Although there were some 
areas in which Annenberg schools improved, there were also a number in which no 
improvement took place, or in which there was initial improvement that was not 
sustained over time. In almost every instance, changes among Annenberg schools 
reflected those across the system as a whole. In general, then, the findings indicate 
that the Challenge made little difference in the long-term school improvement of the 
large number of schools it supported, although it was somewhat more successful in 
the case of the Breakthrough Schools. 

Improvement in Annenberg schools was assessed according to the development 
of numerous measures of the Essential Supports from baseline years of 1994 or 1997. 
Changes among Annenberg schools are summarized as follows (see also Table 12 and 
Table 13): 



108 



The Chicago Annenberg Challenge: Successes, Failures, and Lessons for the Future 



■ The overall quality of classroom instruction improved somewhat, particularly 
teachers’ use of interactive teaching strategies, the intellectual demand of 
instruction, and teachers’ emphasis on writing. Some aspects of student learning 
climate also improved, particularly school safety and classroom personalism. At 
the same time, some small improvements occurred in school leadership, teacher 
professional community, parent involvement in schools, and relational trust. 

■ Concurrently, other areas of the Essential Supports failed to improve or 
weakened. These included student peer support for academic learning, inclusive 
school leadership, and teacher commitment to school. 

■ Initial improvement between 1994 or 1997 and 1999 on a number of Essential 
Supports representing school organizational capacity — school leadership, teacher 
professional community and professional development, parent and community 
involvement, relational trust, and instructional program coherence — was lost by 
2001. Although some measures of organizational capacity were slightly stronger 
or weaker in 2001 than in 1994 or 1997, there was little net change. The 
organizational capacity of Annenberg schools at the end of the Challenge looked 
much like it did at the beginning. 

■ Overall, trends in the development of Annenberg schools followed those in 
demographically similar non-Annenberg schools, indicating little Annenberg 
effect on school improvement. The few initial improvements favoring Annenberg 
schools that began to appear in 1999 disappeared by 2001. There are only two 
exceptions to this general trend. First, teachers in Annenberg schools made less 
frequent use of didactic instruction than teachers in non-Annenberg schools at 
both the beginning and end of the Challenge. Second, by the end of the 
Challenge, Annenberg schools had less instructional program coherence than 
non-Annenberg schools. 

In contrast, the findings indicate that the Challenge’s Breakthrough Schools 
began to develop in ways that distinguished them from other Annenberg schools (see 
Table 14). Although there were no statistically significant differences between 
Breakthrough Schools and other Annenberg schools in student academic and 
nonacademic outcomes or other Essential Supports, Breakthrough Schools sustained 
or strengthened aspects of teacher professional community and, to a lesser extent, 
school leadership and relational trust while other Annenberg schools did not. This 
suggests that these schools may have developed a stronger foundation for subsequent 
development of other Supports, especially instruction and student learning climate 
that, in turn, may promote improvement in student outcomes in the future. 




125 



Part Three: Explaining the Findings 



109 



Achieving the Challenge’s Goals 

Given these findings, what conclusions can be reached about the extent to which the 
Chicago Challenge achieved its goals for school development and improvement in 
student learning? To what extent did the Challenge “[enhance] learning for all 
students through dramatically improved classroom practice”? To what extent did it 
promote the “rethinking] and restructuring of the] basic elements of schooling” in 
the city? To what extent did it address the problems of “time, size, and isolation,” 
promote stronger school-community relationships, and encourage whole school 
change? 

One could argue that the question of goal attainment is not a fair one to pose. A 
case could be made that the Challenge set impossible goals for itself, that it laid out 
an unachievable agenda. A case could be made that the Challenge’s stated goals were 
intentionally rhetorical, not made to set benchmarks for determining its success or 
failure but to draw attention to and mobilize support for a particular vision of 
educational reform. It could also be argued that the Challenge never had a chance to 
develop and work as was intended because the 1995 reform altered dramatically the 
context and support it had assumed and relied upon. 

Regardless of whether the Challenge should be assessed according to its stated 
goals, it is reasonable to consider the extent to which it contributed to the 
development of the schools it supported and to the outcomes of their students. But 
even the question of contribution is difficult to answer — it is hard to disentangle the 
effects of the Challenge from the wider constellation of influences on schools, 
including system-level programs and policies, other improvement initiatives 
implemented in Annenberg schools, new human and fiscal resources that schools 
may have acquired, and so on. Even though this is a complicated matter, this 
research produced a number of clear findings about the development of Chicago 
Annenberg schools as a whole and these findings lead to three general conclusions. 

First, the Challenge contributed in meaningful ways to the development of a 
number of individual schools. Some schools clearly benefited from the extra resources 
the Challenge provided and from their relationships with other schools in their 
networks and their External Partners. As shown in Part Two and in earlier technical 
reports of this research project, there are numerous examples of such benefits. 58 For 
instance, the Challenge provided resources to support leadership training for teachers 
and school administrators; collaborative planning and development activities within 
and across schools to promote teacher professional community; workshops and other 



59 See Wenzel et al. (2001) and Newmann and Sconzert (2000). 
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educational opportunities to strengthen parent involvement and support of student 
learning at home; the introduction of new curricula and programs of instruction, 
particularly in reading and writing; and numerous opportunities for teacher 
professional development aimed at instructional improvement. 

Second, the Challenge seems to have achieved some relative success in promoting 
development in its Breakthrough Schools, specifically in elements of teacher 
professional community and to a lesser extent in leadership and relational trust. 
According to the logic of the Chicago Challenge and the Model of Essential 
Supports, these aspects of school organization serve as building blocks that support 
development of instruction and student learning. Whether the progress achieved in 
the Breakthrough Schools was sustained and built upon after 2001 is an important 
question and one that is left for future inquiry. 

Third, despite contributions to the development of a number of individual 
schools and despite some relative success among Breakthrough Schools, the findings 
provide litde evidence of an overall Annenberg effect on school development or 
student outcomes across the schools it supported. No overall differences were found 
between Annenberg and demographically similar non-Annenberg schools in student 
achievement or the other student outcomes that were examined. With few exceptions, 
the patterns of development found among Annenberg schools were similar to patterns 
of development among non-Annenberg schools. Although Annenberg schools were 
initially developing at a somewhat stronger rate than demographically similar non- 
Annenberg schools on several measures of school leadership and teacher professional 
community, those advantages were lost. At the end of the Challenge, Annenberg 
schools as a group resembled similar non-Annenberg schools on virtually every 
measure of the Essential Supports. 



Explaining the Challenge’s Successes and Failures 

How might these Findings be explained? What factors might account for the lack of 
an overall Annenberg effect? What might explain the pattern of initial improvement 
and subsequent regress in different areas of school development? What factors might 
have contributed to the relative success of the Breakthrough Schools compared to 
other Annenberg schools? There are numerous possible answers to these questions. 
Although those presented below are largely speculative, they are not uninformed. 
They are consistent with the literature on educational innovation and school change. 
Moreover, during the course of the research, a good bit of evidence was collected that 
helps to explain the Challenge’s successes and failures. 
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Lack of Overall Effect 

The failure of the Chicago Annenberg Challenge to achieve an overall effect on 
school development and student outcomes may be explained by at least four different 
factors: (a) shortcomings in the design and implementation of the Challenge; (b) lack 
of capacity among the External Partners to promote school development; (c) lack of 
ability and commitment among schools to engage in the work of the Challenge; and 
(d) lack of external support and “countervailing system forces” that detracted from or 
conflicted with schools’ efforts to develop through the Challenge. Each of these are 
discussed separately; however, it is important to note that it is quite likely that they 
all contributed in some way to compromise the Challenge’s success and that other 
factors were at work as well. 



Shortcomings in Design and Implementation 

Three general shortcomings in the design and implementation of the Chicago 
Challenge might explain, at least in part, the lack of an overall Annenberg effect. 
These include the breadth of the Challenge’s goals and the vague nature of its 
strategy for school development; the inadequacy of resources to support school 
development; and the general weakness of levers for change, particularly the lack of 
accountability. 

Broad Coals, Vague Strategies. The literature on educational change makes 
clear that the implementation of innovations, programs, and policies is enhanced if 
goals and the means to achieve them are made clear to those who must implement 
them . 59 Because of its commitment to the principles of local autonomy and self- 
determinism, the Challenge eschewed the articulation of specific goals and means for 
development. Instead, it laid out broad and diffuse goals that were perhaps overly 
ambitious and rhetorical. Rather than concrete ends, it provided a “vision” and a set 
of general principles for reform. It identified certain priorities around which schools 
should organize their efforts, but offered little direction on how to address them. Nor 
did it specify particular activities or processes for schools to follow. 

Even if there is no best way to promote school development, the Challenge 
provided no particular “theory of change” to guide schools toward more effective 
improvement strategies and away from less effective ones. Instead, its “theory” of 
local self-determinism assigned responsibility for forming specific improvement goals 
and action plans to local school communities, networks, and their External Partners, 



59 Fullan (2001). 




12 & 



112 



The Chicago Annenberg Challenge: Successes, Failures, and Lessons for the Future 



which resulted in substantial variation in the primary foci of network activity and 
school development. As reported in Part One, slightly more than half of Annenberg’s 
networks focused primarily on curricular and instructional improvement. Sixteen 
percent focused on improving student learning climate and social services for 
students and families, and 13 percent were concerned primarily with parent and 
community support and development. The remaining 16 percent adopted more 
comprehensive foci to develop a number of related areas, including curriculum and 
instruction, school leadership, student learning climate, and parent and community 
support. Within these general foci were a large number of specific initiatives such as 
parent education programs, literacy initiatives, programs to integrate the arts and 
technology into the curriculum, health/science education initiatives, support of small 
schools, middle-school restructuring, principal and teacher leadership development, 
and development of stronger relationships between schools and their communities. 

Despite the vagueness of the Challenge’s overall goals and the wide variety of the 
networks’ foci and improvement activities, most principals at Annenberg schools 
perceived with some clarity what their own networks were trying to accomplish. 
According to 1997 survey data, 37 percent of Annenberg principals strongly agreed 
and 59 percent agreed that their networks had clear goals. In 2001, principals’ 
perceptions of goal clarity remained strong. That year, 41 percent of Annenberg 
principals strongly agreed and 57 percent agreed that their networks had clear goals. 

Although most Annenberg principals understood their network’s goals, the 
means by which they pursued them varied substantially across schools, networks, and 
External Partners. Clearly, some strategies were more effective than others. For 
example, this project’s two External Partners repons document and distinguish 
between more and less effective strategies for promoting school development. 60 The 
field research also documented differences in change strategies among individual 
schools. The analyses of developing and nondeveloping schools presented at the end 
of Part Two highlighted some of these differences and their varying degrees of 
effectiveness. 

In general, some Annenberg schools, networks, and External Partners defined for 
themselves relatively effective strategies and others did not. And, while the Challenge 
provided some measure of feedback and guidance, it was constrained in this regard 
by the sheer number of schools and networks it supported. It may have been 
constrained also by its assumptions about the inherent value and effectiveness of local 
initiative and by its reluctance to violate those assumptions by promoting an 
overarching strategy for improvement. These possibilities relate to two additional 
matters discussed later in this section — the ability of Challenge staff to provide 
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adequate professional support and assumptions about the capacity and commitment 
of local schools and Partners to participate effectively in the Challenge and 
implement its particular approach to reform. 

Too Few Resources for Too Many Schools. A second problem in the design 
and implementation of the Chicago Challenge was its scale and the inadequacy of its 
resources. While there is substantial debate about the relationship between funding 
and school effectiveness, most researchers agree that school improvement costs some 
amount of money and other resources. Michael Fullan and Matthew Miles argue that 
change is “resource hungry” because of what it represents — “developing solutions to 
complex problems, learning new skills, arriving at new insights, all carried out in a 
social setting already overloaded with demands.” 61 They continue that such personal 
and collective development “necessarily demands resources.” 

How much money, time, and energy it costs to improve a school is not clear and 
estimates vary. For example, Karen Louis and Matthew Miles found that an average 
principal with a schoolwide reform project spent 70 days, or nearly one-third of her 
time a year, on change management. 62 Teachers most closely engaged with the 
change effort spent some 23 days a year, or 13 percent of their time, on reform. 

Louis and Miles found that “serious” change in large urban high schools required an 
annual investment of between $50,000 and $100,000 (in late-l980s dollars). Others 
have argued that the cost of implementing comprehensive whole-school reform, 
while varying greatly from initiative to initiative, may cost more than $160,000 per 
year with first-year costs (which may include one-time costs of training and 
materials) ranging between $100,000 to $350, 000. 63 Although Fullan and Miles note 
that how schools spend money is the most important determinant, they conclude 
that a minimum level of stable funding is always needed to support change. Lack of 
resources has been found to be a common problem for schools trying to implement 
comprehensive whole-school models of reform. 64 Moreover, the literature shows 
clearly that implementation weakens or ceases when resources that initially fueled the 
reform are no longer available. 65 

In general, the Chicago Challenge provided too few resources and too little 
support to too many schools and External Partners. Although it never claimed that 
its grants were intended to “purchase” improvement or that they were even large 
enough to do so, they were intended to “stoke” development through the facilitation 



61 Fullan and Miles (1992), p. 750. 

62 Louis and Miles (1990). 

63 Keltner (1998) and Odden (1997). 

64 Smith et al. (1997); see also Murphy and Datnow (2003). 

65 See Glennan (1998). 
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of relationships between schools and Partners and to lever additional resources to 
support those relationships. In some schools, Challenge resources clearly made an 
important contribution to local development efforts. Overall, however, the Challenge 
spread its resources thinly across the 210 schools and 45 External Partners it 
supported. Even at the peak of its network funding, the Challenge made relatively 
low monetary investments in local improvement efforts. In 1999, it provided schools 
in implementation networks on average about $47,000 in money and services 
through their External Partners. This amount was about 1 percent of a typical 
elementary school's operating budget. By the end of the Challenge in 2001, these 
modest investments had been reduced to virtually nothing. That year, the amount of 
money provided to local schools averaged about $2,600. 

Similarly, it would have been very difficult for Challenge staff to provide 
adequate professional support to all of the schools and External Partners that received 
funding for the many different local initiatives in which they engaged. To be sure, it 
organized numerous workshops for schools and Partners. It established principal 
support groups and sponsored fairs for schools and Partners to share their work and 
accomplishments. Some of the schools and Partners received direct feedback and 
support from Challenge staff. As noted in Part One, the primary responsibility for 
providing such professional support fell to one staff member-the Program Director 
who was joined in this work by a Grants Manager and the Challenge’s Executive 
Director. Both the Program Director and the Grants Manager had some but not 
extensive experience in school development. The Executive Director was from the 
local foundation community and his primary experience was in grant making and 
community development. It is difficult to see how a staff of this size could provide 
the guidance, feedback, and support that would be required by 210 schools and 45 
External Partners no matter how much experience it had in school development. 

Perhaps because of the need to find the right economies of scale, Challenge staff 
devoted much of its time to working with External Partners rather than working 
directly with individual schools. This is not to say, of course, that it did not provide 
any assistance to individual schools or groups of schools — it did. Still, because of the 
sheer number of schools it funded, the Challenge may have considered it more 
efficient and effective to focus its efforts on the External Partners, operating from the 
logic that through them it could reach more schools than if it tried to work with 
schools directly. 

As discussed in the research project’s second report on Annenberg External 
Partners, the Partners generally had positive views of the support they received from 
Challenge staff, particularly the workshops that the Challenge sponsored and the 
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opportunities it provided for them to interact and learn from one another. 66 
Specifically, the Partners that were interviewed cited as helpful the Challenge’s 
workshops on media relations and data-driven decision making, in addition to the 
workshops that brought in various outside speakers. They also considered helpful the 
individual attention they received from members of the Challenge staff, including the 
Executive Director. 

The Partners that were interviewed also noted certain weaknesses in the support 
they received, weaknesses that could be attributable in part to the breadth of the 
Challenge’s goals and the small size of its staff. Some said that while they found the 
workshops helpful, they thought that the Challenge could have provided more 
opportunities for partner-to-partner sharing and more follow-up activities on 
workshop topics. They also expressed disappointment that they received little 
feedback on the reports of network activity that they were required to file twice a year 
(they almost universally complained about the burdens these reporting requirements 
placed on them). They saw such feedback as a potentially valuable source of learning 
for improving their work with schools. And while they considered their individual 
relationships with Challenge staff helpful, many expressed frustration with the lack of 
time staff members had for on-site school visits. They understood that the small size 
and the wide-ranging responsibilities of the Challenge’s staff made it difficult for 
them to become very involved at the school level. Still, they said that they had hoped 
for more. 

The principal surveys provided additional evidence of weaknesses in staff support 
at the school level. Most Annenberg principals reported that Challenge staff usually 
made themselves available to support network activity; in 1997, 18 percent of 
principals strongly agreed and another 69 percent agreed that Challenge staff 
members were usually available to support their networks. In 2001, these proportions 
were virtually the same — 85 percent of Annenberg principals agreed or strongly 
agreed that Challenge staff members were available to support their networks. At the 
same time, 52 percent of Annenberg principals reported in 1997 that Challenge staff 
had no real effect on their schools’ network activities. In 1999, the last year that this 
question appeared on the principal survey, there was a decline in the staff s reported 
influence — 63 percent of Annenberg principals reported that Challenge staff had no 
real effect on their network activities. 

It is also important to recall that the Challenge considered the External Partners 
to be important resources for local school improvement (further discussion of 
External Partners appears below). Here it is important to note that one role that could 
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have been performed by External Partners was to help schools secure additional 
external resources and services to support their development initiatives. Evidence from 
the principal surveys suggests that some Partners were more effective in this regard than 
others. In 2001, 35 percent of Annenberg principals strongly agreed that their External 
Partners were able to help their schools acquire needed services and resources. Fifty- 
seven percent of principals agreed that their Partners provided some such assistance. 
Ten percent disagreed or strongly disagreed that their Partners were helpful in this 
regard. No data are available to assess the usefulness or adequacy of such services and 
resources; the available evidence only points to variability in principals’ views of their 
Partners’ help in securing them. 

Weak Levers for Change. The literature on educational reform emphasizes the 
importance of developing new knowledge, skills, and commitments necessary for 
change, but it also emphasizes the importance of incentives and accountability for 
participation in change processes, for applying new knowledge and skills, and for 
incorporating change into routine practice. 67 The Chicago Challenge was cognizant of 
the need to promote the development of new knowledge and skills, to provide 
incentives, and to hold schools and networks accountable for the resources they 
received. Overall, however, while the Challenge recognized the importance of each of 
these “levers” for change, none were particularly well developed or particularly strong 
and sustained. 

The Challenge stressed teacher professional development and provided a number 
of opportunities for principals to develop new knowledge and skills to support their 
schools development. As reported in Part Two, it achieved some success in this 
regard. In both 1997 and 1999, participation in professional development activity 
across Annenberg schools was significantly greater than participation across 
demographically similar non-Annenberg schools. By 2001, however, these differences 
disappeared. 

The Challenge also introduced some measures of accountability to promote 
school participation and change. For example, after initial rounds of network 
funding, it decided not to renew grants to particularly weak networks and External 
Partners. In later rounds of funding, it worked directly with schools and Partners to 
increase the overall quality of their proposals and plans for school improvement. 
Despite these and other such efforts, the Challenge’s overall design provided few 
mechanisms to lever very much change. As noted above, it provided few financial 
resources to schools and Partners. Moreover, what little funding it did provide 
through 1999 was greatly reduced by 2001. Although some of the Partners that were 
interviewed thought that Challenge grants provided some leverage in working with 
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their schools, it is likely that the sizes of the grants were simply not large enough to 
command much attention or instill a strong sense of accountability among such a 
large number of schools. 

There was little evidence of any real or perceived consequences among schools 
and Partners for failing to participate actively in network or Challenge-sponsored 
activities. The Challenge’s only real accountability mechanism was the threat of 
discontinuing its financial support to schools and Partners. Even though it spent 
substantial time and resources to monitor school and Partner activity and the 
expenditures of its funds, there is not much evidence that the Challenge was able to 
create a strong sense of imperative for participation in Annenberg activity or for 
change across the large number of schools it funded. Moreover, the Challenge may 
have weakened its own hand by making an implicit commitment to continue to 
support the networks and Partners it coached in the proposal development process. In 
the least, this most likely reduced any real or perceived threat that it would withdraw 
resources from the activity it helped schools and the Partner develop. Given the rather 
small amount of money provided, losing Annenberg money was probably of little 
consequence to most schools or Partners. Regardless, the threat was rarely exercised. 
Virtually all of the schools that received funds in 1998 kept receiving them until the 
end of the Challenge, albeit in diminished amounts. 



Lack of Capacity among External Partners 

This research project did not set out to study directly the capabilities and resources of 
External Partners. Nonetheless, a good bit of evidence points to how differences in 
Partners’ experience and expertise may have affected their ability to support 
improvement among the schools with which they worked. An earlier study of 
Annenberg External Partners found that most Partners achieved variable success with 
schools in their networks. 68 That is, most were more successful with some schools in 
their networks than with others. 

Notwithstanding that substantial proportions of Annenberg principals reported 
on surveys that their External Partners were a source of impetus and support for 
change, there is reason to believe that Partners varied substantially in their 
knowledge, understanding, and ability to effectively promote school improvement. 
First, not all Partners entered the Challenge with experience in working with schools. 
One-third had no experience before 1995 in working with schools on long-term 
improvement projects. Second, the primary network foci and activities proposed by 
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Partners for funding reveal wide variation in thinking about how school development 
is achieved. As discussed earlier, most set agendas focused on developing single 
Essential Supports without attending to the development of other complementary 
supports (e.g., development of classroom instruction without attention to 
development of teacher professional community, parent involvement, and school 
leadership to support it). Only one of six External Partners pursued a more 
comprehensive development agenda to develop in a coordinated manner two or more 
related Essential Supports. Research suggests that attention to the systemic 
relationships among different aspects of school organization and practice is most 
conducive to school development. 69 Indeed, the Challenge recognized the problem of 
External Partner capacity early on. As noted in a previous research project report, over 
the course of the initiative, the Challenge became more intentional in its grant making 
and provided professional support to grant applicants, most of whom were External 
Partners. 70 This move was partly in response to what it perceived was a lack of 
imagination in the grant proposals it received and funded. It was also in response to 
concerns voiced by grantees who seemed to be unclear about the Challenge’s 
expectations and principles of reform. The Executive Director observed that Challenge 
staff“[had] not been highly impressed with the creativity and inventiveness” of the 
implementation proposals funded in the first round of grant making. In 1997, he 
observed of External Partners, 

We realized [that] just because you build it they will not come....[W]e had to add a 
strong program resource piece.... We [now] say, "Not only do we want to lure you 
into these relationships [with schools] with the money, ...but we also need to lead 
you or expose you to a set of resources.” We needed to inject ideas... with the spirit 
that. . .we’re still respecting your choice. 

It is not clear whether the approaches to school improvement that External 
Partners pursued were a function of their understanding and “theories” of school 
change (or lack thereof) or to other factors. It is certainly possible that the relatively 
few financial resources the Challenge provided may have constrained the 
ambitiousness of their work and made it difficult to engage in more creative and 
systemic school development activity. For some Partners, Annenberg grants were a 
substantial portion of their budgets. For others, it was an important but relatively 
small amount. Regardless, it was not a lot of money for any Partner to work with all 
the schools in their networks. The research project’s first report on Annenberg 
External Partners noted that many Partners did not have enough staff members to 
work with the schools in their networks. 71 That report also observed that some 
External Partners hired teachers from their network schools to work as professional 
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development leaders or curriculum coordinators. In some instances, this had the 
unintended consequence of exacerbating leadership shortages in the schools from 
which these teachers were hired and thereby undermining the Partners' work in those 
settings. 



Lack of School Capacity to “Do Annenberg” 

Thomas Timar and David Kirp have argued that the success of school reform 
initiatives depends in significant ways on the capacity of schools to engage in and 
implement those initiatives effectively . 72 In their words, schools need the 
“institutional competence” to fulfill the demands of reform. By institutional 
competence, Timar and Kirp refer to the aspirations, commitments, and norms of a 
school that direct its work and its efforts to improve. They also refer to the 
knowledge and skills of teachers and administrators to respond to the reform, 
implement it, and achieve its objectives. 

Others have made the same general argument . 73 Gene Hall and Shirley Hord 
have pointed out the importance of a school's state of “readiness” and its initial 
commitments to an innovation and to that innovation's long-term implementation 
and effectiveness . 7 ' 1 Matthew Miles has argued that successful school change most 
often requires schools to possess the ability to do good “problem coping .” 75 In Miles' 
view, school improvement is not always rational or predictable. Schools need the 
ability to locate, analyze, and address problems that are inevitably part of the 
improvement process. Miles extends this argument in work with Michael Fullan and 
Karen Louis, reasoning that because change has “no blueprints” and because rational 
planning models for complex social change like education reform do not work, 
schools need the collaborative capacity for analysis, incremental decision making, and 
experimentation . 76 They also need the capacity to develop normative consensus 
around the improvement effort. Moreover, because change initiatives do not manage 
themselves, schools must have the capacity to manage them well. At a minimum, this 
requires that groups responsible for implementation have the ability to collaborate, 
solve problems, and make decisions together. 

A primary organizing theme of the Chicago Challenge was the empowerment 
and self-determinism of local actors, members of school communities working in 
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networks with External Partners, to improve their own schools. Closely related to 
this theme of local initiative and self-determinism was the theme of capacity 
building. The Challenge called on the schools and Partners it supported to build 
organizational capacity by addressing the issues of time, size, and isolation. It also 
challenged schools and Partners to build capacity for instructional improvement 
through teacher professional development. And so on. At the same time, it made 
certain assumptions that schools and Partners already possessed some requisite 
capacity to engage in decentralized, self-determined, collaborative work for local 
capacity development. 

A case can be made that in order for schools to have successfully engaged in the 
Challenge’s “style” of reform, that is, for schools to “do Annenberg” well, they would 
have needed to possess some base of human, social, and material resources to support 
collaborative development work within schools, among schools, and with External 
Partners. This base of resources might well have consisted of inclusive collaborative 
leadership, strong working relationships among teachers, and strong relationships 
between the school and parents. Schools would also have needed some base of 
commitment to the Challenge and to its approach to reform. Indicators of such 
commitment might well have included the alignment of Annenberg’s goals with the 
school’s own goals for improvement, the school giving priority to the Annenberg 
initiative over other initiatives, and committing people and time to make the effort 
work. 

As noted earlier, the Challenge recognized that External Partners varied 
considerably in their understanding of its concepts and principles. It recognized 
variability in the imagination, creativity, and potential effectiveness of the proposals 
they submitted. And, as described in Part One and as will be discussed later in this 
section, it introduced a different funding strategy with the Breakthrough School 
initiative, a strategy that was more intentional in considering the development and 
capacities of the schools it funded. At the same time, there is little evidence that the 
Challenge systematically considered the capacity of schools to “do Annenberg” 
between 1995 and 1998 when it awarded the majority of its implementation grants 
and committed most of its funds. 

Be that as it may, it is worthwhile to examine the capacity of Chicago Annenberg 
schools to engage productively in the Challenge’s approach to reform. First, how 
schools scored at the beginning of the Challenge on different indicators of human, 
social, and material resources is examined. This is followed by an examination of 
different indicators of school commitment to participate in the Challenge. Finally, 
another side of the capacity issue is explored; that is, the sources of disruption and 
persistence that may have compromised a school’s ability to improve through the 
Challenge’s approach to reform. The evidence indicates that the Challenge 
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supported a substantial number of schools with relatively weak capacity to engage in 
its approach to reform. Coupled with potentially strong internal sources of 
disruption and persistence, such weakness may help explain the lack of an overall 
Annenberg effect on the development of schools the Challenge supported. 

Human, Social, and Material Resources. The citywide principal surveys asked 
Annenberg principals whether the Challenge provided resources that were useful for 
their schools’ development. They also asked principals whether their schools had 
enough of their own resources — staff, time, and other resources — to make 
participation in the Challenge “pay off.” Their responses reveal substantial variation. 
In both 1997 and 2001, more than 90 percent of Annenberg principals agreed or 
strongly agreed that participation in the Challenge provided their schools with useful 
resources. At the same time, they were divided in their assessments of the adequacy of 
the resources their own schools possessed to make participation worthwhile. In 1997, 
45 percent of Annenberg principals disagreed or strongly disagreed that their schools 
had enough resources to make such a difference. In 2001, roughly the same 
percentage of principals considered their schools’ resources inadequate. 

Data from the 1997 teacher surveys were used to assess the strength among 
Annenberg schools of key human and social resources that might be used to support 
school development through participation in the Challenge. Measures of several 
aspects of school leadership, school orientation toward innovation, teacher 
professional community, relational trust, and school relationships with parents were 
examined. These 1 997 data provide a picture of school capacity at the start of the 
Challenge, capacity that might have supported Annenberg work and provided a base 
on which to build additional capacity. 

As shown in Table 18, in 1997, substantial proportions of Annenberg schools 
scored in the weakest categories of measures of human and social resources to support 
school development (see Appendix E for specific definitions of measure categories). 
That year, 17 percent of Annenberg schools reported minimal and limited 
orientation to innovation. In these schools, relatively few teachers were reported to 
try new ideas and take risks to improve their practice. There was substantial 
disagreement that teachers were continually learning, were encouraged to grow, and 
had a “can do” attitude. No teachers or only some of the teachers in these schools 
were reported to try new ideas and take risks to improve their practice. Substantial 
percentages of Annenberg schools also reported weaknesses in various aspects of 
school leadership. Twenty-four percent of Annenberg schools reported minimal 
principal support for change. In these schools, some teachers agreed and some 
disagreed that their principals encouraged them to try new methods, were willing to 
make changes, and provide strong support for changes introduced at the school. 
Teachers in these schools disagreed that their principals encouraged them to take 
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risks, provided adequate professional development for changes introduced at their 
schools, and involved teachers in such change initiatives. In addition, 18 percent of 
Annenberg schools reported weaknesses in principal instructional leadership. In these 
schools, some teachers agreed but others disagreed that their principals made 
teaching expectations clear, set high standards for both teaching and student 
learning, and communicated a clear vision for their schools. Teachers disagreed that 
their principals pressed them to implement what they learned in professional 
development activities, understand how students learn, and track students’ academic 
progress. In 24 percent of Annenberg schools, teachers reported that their principals 
promoted parent and community involvement but they disagreed that their 
principals worked to create a sense of community in their schools or were committed 
to shared decision making. Finally, 36 percent of Annenberg schools reported 
weaknesses in joint problem solving. While teachers reported general openness of 
expression in their schools, they indicated that problems and conflicts were often 
ignored or avoided. 

Substantial percentages of Annenberg schools also reported weaknesses in 
different aspects of teacher professional community in 1997. One-quarter of 
Annenberg schools reported limited levels of peer cooperation and collaboration. A 
similar percentage of schools reported that teachers engaged only occasionally in 
reflective dialogue about their teaching. About one-third of Annenberg schools 
reported weak focus on student learning and a very limited sense of teacher collective 
responsibility for student learning and school improvement. Substantial percentages 
of schools also reported weaknesses in relational trust. Twenty-one percent of 
Annenberg schools reported minimal levels of trust between teachers and principals. 
More than half of the schools reported no trust or minimal trust among their 
teachers. Forty-two percent reported minimal levels of trust between teachers and 
parents. Finally, 39 percent of Annenberg schools reported minimal and limited 
levels of parent involvement. 

It is important to recall that some of these human and social resources grew 
stronger in Annenberg schools between 1997 and 1999 (e.g., inclusive leadership, 
principal instructional leadership, focus on student learning, teacher-parent trust). In 
addition, teacher participation in professional development activity and the quality of 
the professional development improved among Annenberg schools during this time. 
In almost all cases, however, these improvements were lost after 1999. 

Time is another resource that is necessary to engage productively in reform. Both 
Annenberg principals and External Partners noted throughout the course of this 
study that although the Challenge made this a priority in its reform agenda, lack of 



Part Three: Explaining the Findings 



123 



time was a persistent impediment to their school development efforts. Their 
observations are consistent with the literature . 77 Most reform initiatives add new 
work but few alleviate other responsibilities or provide additional time for planning 
and implementation. Recent studies of whole school comprehensive reform models 
have found that insufficient time for planning, collaboration, and professional 
development is a common barrier to the implementation of these models and a 
frequent concern for teachers . 78 



Table 18. Percentages of Chicago Annenberg Schools and the Weakest Categories in Which They 
Scored on Measures of Human and Social Resources to Support Development, 1997 



Measures 


Categories 


Percent of Schools 


Orientation toward Innovation 


Minimal and limited 


17% 


Principal Support for Change 


Minimal 


24% 


Principal Instructional Leadership 


Mixed 


18% 


Inclusive Leadership 


Mixed 


24% 


Joint Problem Solving 


Weak 


36% 


Peer Collaboration 


Limited 


25% 


Reflective Dialogue 


Occasional 


24% 


Focus on Student Learning 


No focus and not very focused 


31% 


Collective Responsibility 


Very limited and limited 


33% 


Teacher-Principal Trust 


Minimal 


21% 


Teacher-Teacher Trust 


None and minimal 


54% 


Teacher-Parent Trust 


Minimal 


42% 


Parent Involvement 


Minimal and limited 


39% 



77 Elmore and McLaughlin (1988) and Smith (2000). 

78 Muncey and McQuillan (1996); Ross et al. (1997); and Smith et al. (1997). 
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As noted earlier, substantial proportions of Annenberg principals reported on 
both 1997 and 2001 surveys that their schools lacked the staff, time, and other 
resources to make participation in network activity really “pay off.” The research 
project’s first technical report observed that six of the 12 principals that were 
interviewed identified lack of time as a specific impediment to their schools’ 
participation in network activities during their first year of funding. 79 Each pointed to 
lack of time as a reason for not attending network meetings themselves or for not 
sending a member of their staff. The research project’s first report on External 
Partners observed that the structure of the normal school day, combined with the 
number of programs requiring teacher participation, restricted teachers and 
administrators’ time to focus on the Partners’ programs. 80 The project’s second 
technical report on the development of Annenberg schools also identified lack of 
time as an impediment to teacher participation in Annenberg activity and to their 
efforts to experiment with and implement new ideas they learned from that activity. 81 

Commitment. Another important element of a school’s capacity for reform 
consists of the various commitments that support reform implementation. There are 
numerous indicators of these commitments, including the compatibility of the 
reform’s goals and the school’s own goals for improvement, the centrality of the 
reform to the school’s work, and the allocation of people and time to the reform 
effort. The data point to substantial variation among Annenberg schools on several of 
these indicators. 

Most principals reported that their Annenberg network’s goals coincided at some 
level with their own school’s goals for improvement. In 1997, 3 4 percent of 
Annenberg principals strongly agreed and 60 percent agreed that their schools’ goals 
were aligned with those of their networks. Only 7 percent disagreed or strongly 
disagreed. In 2001, somewhat greater proportions of Annenberg principals saw 
alignment — 38 percent strongly agreed and 61 percent agreed while only 2 percent 
disagreed. 

Although reports of goal alignment were strong, there was more variability in the 
degree to which Annenberg principals saw the Challenge as central to their schoofs 
work. In 1997, 21 percent strongly agreed and 49 percent agreed that Annenberg 
activities were central to their school’s work but 31 percent disagreed or strongly 
disagreed this was the case. Although principals were somewhat more positive about 



79 Smylie et al. (1998). 
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the importance of Annenberg activity in 2001, variability remained. That year, 34 
percent of Annenberg principals strongly agreed and 52 percent agreed that the 
Challenge was central to their school’s work while 14 percent disagreed. 

Teacher participation is another indicator of school commitment. Indeed, the 
commitment of people and time to participate in Annenberg activity may be more 
important than goal alignment and centrality. Data from the principal surveys 
indicate that teacher participation in Annenberg activity varied substantially among 
schools (see Table 19). In 1997, 30 percent of Annenberg principals reported that 
less than 20 percent of their teachers participated in Annenberg activity; 17 percent 
reported teacher participation rates of between 21 and 40 percent; 12 percent 
reported participation rates of between 41 and 60 percent; 17 percent reported rates 
of between 61 to 80 percent; and the final 15 percent of principals reported that 
more than 80 percent of their teachers participated in Annenberg activity. In 2001, 
Annenberg principals reported somewhat higher rates of teacher participation, but 
substantial variation remained among schools — 18 percent reported that less than 20 
percent of their teachers participated in Annenberg activity; 15 percent reported 
teacher participation rates of between 21 and 40 percent; 25 percent reported rates of 
between 41 to 60 percent; 18 percent reported rates of between 61 and 80 percent; 
and 23 percent reported that more than 80 percent of their teachers participated. 

There was also substantial variation in the frequency with which teachers 
participated in Annenberg activity (see Table 20). Thirty-seven percent of principals 
reported on the 1997 survey that their teachers participated on average about once a 
month or less in Annenberg activity. Twenty-three percent of principals reported 
that on average their teachers participated two to three times a month. Sixteen 
percent reported weekly teacher participation and 24 percent reported participation 
more than once a week. In 2001, principals reported lower rates of teacher 
participation. Half reported that their teachers participated on average about once a 
month or less in Annenberg activity. Thirty-three percent reported that teachers 
participated two to three times a month. Three percent reported weekly participation 
and 15 percent reported participation more than once a week. 



Table 19. Percentage of Annenberg Principals Reporting the Proportions of Teachers at Their Schools 
Who Participated in Annenberg Activities, 1997 and 2001 



PROPORTIONS OF TEACHERS 


1977 


2001 


Less than 20% of teachers 


31% 


18% 


21 to 40% of teachers 


17% 


13% 


41 to 60% of teachers 


12% 


25% 


61 to 80% of teachers 


17% 


18% 


More than 81% of teachers 


22% 


23% 
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Table 20. Percentage of Annenberg Principals Reporting Different Frequencies of 
Teacher Participation in Annenberg Activities, 1997 and 2001 



FREQUENCY OF TEACHER PARTICIPATION 


1997 


2001 


Once a month or less 


37% 


30% 


Two or three times a month 


23% 


33% 


Once a week 


16% 


3% 


More than once a week 


24% 


15% 



Internal Sources of Disruption and Persistence. In addition to the various 
resources that schools may have to support improvement, they also have sources of 
internal disruption and persistence that may compromise improvement. The 
literature is full of examples of these sources. 82 Studies of educational innovation find 
that the loss of key leaders and staff turnover make it very difficult to sustain 
commitment to change initiatives over time. 83 Often the structure of work and time 
makes it difficult for teachers and administrators to participate in improvement 
activity and implement changes in their practice. 84 Research on school organizational 
cultures and the institutions of schooling concludes that norms, beliefs, rituals and 
ceremonies, symbols, political relations, and work rules and relationships form 
routines or “scripts” that give meaning and govern much of what takes place in 
schools. 85 These scripts, which are often unspoken and taken for granted, are strong 
sources of organizational stability and often make persistence prevail over attempts to 
change. There is some evidence that such sources of disruption and persistence 
existed in many of the schools the Challenge supported. To the extent that they did 
exist, they may have contributed to the failure of the Challenge to achieve an overall 
effect. 

There is some evidence that Annenberg schools as a group experienced 
substantial staff turnover and such turnover may have made it very difficult to make 
and sustain much development. Surveyed Annenberg principals reported that they 
hired an average of 25.2 new teachers between 1995 and 2001 or 3.6 teachers per 
year. Between 1999 and 2001, they reported hiring an average of 9.7 new teachers or 
3.2 teachers per year. According to CPS, the average size of an elementary school 
faculty is 26.3 teachers. While the number of new teachers hired is not necessarily the 
best indicator of turnover (it doesn’t take into account the numbers of teachers who 
leave a school), it does give some idea of the change or “churn” in personnel taking 
place. A rough estimate indicates that the number of new teachers hired by 



82 Fullan (2001); Miles (1993); Smylie and Crowson (1996). 

S3 Fullan (2001). 

84 Smith (2000). 

85 DiMaggio and Powell (1991), Firestone and Louis (1999); Rowan and Miskel (1999); Schein 
(1992); and Zucker (1987). 
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Annenberg schools between 1995 and 2001 represented on average between 12 and 
14 percent of their entire faculties each year. Data from CPS personnel records 
indicate that these principal reports underestimated the numbers of teachers who 
came to their schools each year. For example, these records indicate that on average, 
for the 1997-98 school year (the latest data available for this report), 1 5.7 percent of 
teachers in Annenberg schools were new to their schools that year. This percentage is 
the same as the percentage of teachers in non-Annenberg schools who were new to 
their schools. 

In addition to change in faculty, the loss of key leaders may threaten school 
improvement efforts. Between 1996 and 1999, there were a number of documented 
instances from the field research schools where the loss of curriculum coordinators 
who were hired and trained by External Partners and paid for by Challenge grants all 
but terminated the progress schools were making toward improvement. 86 In other 
field research schools, improvement efforts stalled when the principals who initiated 
them retired or left for other schools. 

Data from teacher surveys and the field research provide evidence that some 
Annenberg schools had cultures that were much more conducive to change than 
others. Recall that in 1997, nearly 20 percent of Annenberg schools ranged from 
“minimal” to “limited” on the survey measure of school orientation toward 
innovation (see Table 18). As described earlier, minimal and limited orientation 
means that teachers reported that about half of the teachers in their schools really try 
to improve their teaching. Some agree and others disagree that teachers at their 
schools are continually learning, are encouraged to grow, and have a “can do” 
attitude. Teachers are likely to report that only some of their peers try new ideas and 
take risks to improve their practice. 

Another indicator of a culture conducive to change is the expectations teachers 
hold for their students’ learning and for their future. It may be very difficult to 
engage schools and teachers in improvement activity if teachers have relatively low 
expectations. That is, if teachers do not think their students can learn or be 
successful, they may also believe that efforts to change schools are for naught. 

Two items on the teacher survey give some indication of the variability of 
expectations that teachers in Annenberg schools held for their students. These items 
asked teachers what percentage of their students they believed would graduate from 
high school and what percentage would go on to attend a two- or four-year college. 



86 Wenzel et al. (2001). 
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In 1997, teachers in Annenberg schools held a range of expectations for their 
students (see Table 21). Some of these expectations were quite low, perhaps 
contributing to school cultures that were not particularly conducive to improvement. 



Table 21. Expectations of Annenberg Teachers for Their Students’ Educational 
Futures, 1997 and 1999 



| PERCENT OF ANN ENBERG TEACHERS 


Percentage of the students I teach who I expect will graduate from high school 


1 to 25 percent 


4% 


26 to 50 percent 


13% 


50 to 75 percent 


35% 


76 to 1 00 percent 


48% 


Percentage of the students I teach who 1 expect will attend a two- or four-year college 


1 to 25 percent 


23% 


26 to 50 percent 


23% 


50 to 75 percent 


21% 


76 to 100 percent 


21% 



The field research revealed that External Partners sometimes encountered school 
cultures that confounded their efforts to promote development. This project’s first 
report on External Partners noted that some Partners confronted a culture of privacy 
and autonomy among teachers that made it difficult to promote joint problem 
solving and collaborative professional development. 87 Several of the Partners observed 
that some teachers with whom they worked believed that since previous reform 
initiatives did not seem ever to be sustained — that they came and went with change 
in school administrators — it was not prudent for them to invest seriously in new 
initiatives. Similarly, the Partners reported that in a number of cases, principals with 
whom they worked were either unwilling to or incapable of infusing Partner- 
sponsored activities into their schools. Some principals appeared to Partners to be 
“victims” of an administrative culture that worked against-the Partners’ efforts for 
school development. Similar evidence was presented in the research project’s 
technical report on the development of Annenberg schools between 1996 and 1999. 88 

Program overload and fragmentation among many improvement initiatives were 
yet other problems that Annenberg schools faced. The literature on school 
improvement has long identified the potential problems that multiple change 
initiatives pose to the effective allocation of teachers’ time and other resources. 89 The 
issues of program overload and fragmentation and the attendant problem of 
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incoherence among programs were examined in detail in a separate report of the 
research project. 90 Here it is useful to highlight some of the evidence that illustrates the 
problems posed by the presence of too many initiatives. 

The effects of too many programs and the competition among them were first 
identified in the research project’s first technical report, Getting Started?' In that 
report, some Partners who were interviewed observed that the work they sought to 
accomplish often competed with other initiatives for teachers’ and administrators’ 
time and attention. In these cases, Partners who tried to promote whole school 
development found they had limited influence. In later interviews, more than half 
reported that their schools were involved in so many programs besides Annenberg “it 
was often difficult to get the schools to pay serious attention to [their] programs.” 92 

The scope of this problem in Annenberg schools is suggested by data from the 
principal surveys. Both 1997 and 1999 surveys asked principals to report the extent 
to which they saw the Challenge as just one of many programs they had at their 
schools and the degree to which their teachers devoted time to Annenberg activities 
as opposed to other projects. In 1997, 77 percent of Annenberg principals agreed or 
strongly agreed that the Challenge was just one of many programs at their schools 
(see Table 22). In 1999, 80 percent of Annenberg principals agreed or strongly 
agreed that this was the case. In 1997, more than half of Annenberg principals 
disagreed or strongly disagreed that of all initiatives at their schools, the most teacher 
time was devoted to Annenberg activities (see Table 23). In 1999, a similar 
proportion made the same assessment. Taken together, this strongly suggests that the 
Challenge faced competition for time, attention, and effort from other improvement 
initiatives in a substantial number of the schools it supported. This competition may 
have made it quite difficult to engage schools in Annenberg activity in a way that 
might lead to significant lasting development. Indeed, this situation may have grown 
worse in the Challenge’s last few years. As noted above, program coherence in 
Annenberg schools declined between 1999 and 2001. 



Table 22. Percentage of Annenberg Principals Who Agreed with the Statement “The Annenberg 
Challenge is just one of many programs we have at this school,” 1997 and 1999 





1997 


1999 


Strongly Agree 


20% 


15% 


Agree 


57% 


65% 


Disagree 


21% 


18% 


Strongly Disagree 


1% 


3% 
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Table 23. Percentage of Annenberg Principals Who Agree with the Statement “Of all external projects, 
most teacher time is devoted to Annenberg activities,” 1997 and 1999 





1997 


1999 


Strongly Agree 


12% 


13% 


Agree 


35% 


37% 


Disagree 


50% 


48% 


Strongly Disagree 


3% 


3% 



Countervailing Forces 

Yet another reason for the failure of the Chicago Challenge to achieve an overall 
effect on school development may have been the influence of “countervailing system 
forces.” This refers to outside influences that pulled schools in directions contrary to 
those promoted by the Challenge and its External Partners. Of course, what one 
considers to be countervailing depends on where one stands. It is certainly 
conceivable that the Challenge itself could have been seen as a countervailing force to 
other reform agendas. Indeed, evidence of such a view among some CPS 
administrators and some of Chicago’s civic elite was presented in a 1999 research 
project report on the Challenge’s establishment and early operation. 93 In this 
discussion, countervailing forces are defined and examined from the perspective of 
the Challenge as factors that worked against or failed to support its work and what it 
sought to accomplish. 

Some of the most visible and potentially powerful sources of countervailing 
influence on Annenberg activities were the school system’s major reform initiatives. 
Part One described the CPS administration’s major initiatives under the 1995 reform 
and the potential areas of conflict with the Chicago Challenge. In this section, 
additional evidence is presented of how such conflicts were perceived and how they 
may have compromised the overall effectiveness of the Challenge. 

First, the problem of conflict among multiple policy initiatives and reforms has 
been discussed in the literature on educational change for some time. In an early 
study of federal Title I programs, Jackie Kimbrough and Paul Hill found evidence of 
conflict in the implementation of core programs and multiple federal categorical 
programs in each of the 24 schools they studied. 94 Their research pointed to the 
difficulty faced by teachers and school-level administrators in managing multiple 
programs, particularly those with conflicting goals and those that competed with each 
other for scarce monetary and human resources. Kimbrough and Hill found that where 
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conflict existed, the implementation of both core school programs and categorical 
programs could be compromised. 

This problem remains part of today’s reform landscape. Timar and Kirp, as well 
as Thomas Hatch, argue that because of the magnitude of today’s reform efforts, 
tensions and conflicts among policies and improvement initiatives are all but 
inevitable . 95 In a recent review of the literature, Michael Knapp and his colleagues 
provide new insight . 96 They examined the convergence of different types of 
educational reform at the school level — decentralized governance and decision 
making; systemic innovations in curriculum, instruction, and student assessment; and 
integrated educational, social, and health services for students. While they found that 
litde empirical evidence has been presented in the literature about the converging 
effects of different initiatives, they argued that research examining each initiative 
separately allowed them to anticipate the effects of convergence on teachers and 
administrators and to anticipate their likely responses. 

Knapp and his colleagues argued that, at a minimum, these reforms would add 
new responsibilities to teachers’ and administrators’ workloads. The reforms would 
increase pressure for collaboration and call on teachers and administrators to form 
different conceptions of professional work and develop new knowledge and skills to 
perform that work well. The reforms would increase demands on teachers’ and 
administrators’ time both in the short and long term and create inevitable compromises 
in how they allocated their time and effort. Knapp and his colleagues predicted that 
when faced with such convergence, teachers and principals would likely respond 
strategically and defensively. They would find ways to cope and to reduce demands to 
manageable levels. Particularly where converging reforms might conflict, but certainly 
where the demands of reform exceeded available time and effort, teachers and 
administrators would prioritize reforms, give them selective attention and, with regard 
to those reforms that conflicted with their own values and practices or threatened 
“better” reforms, engage in organized or passive resistance. Knapp and his colleagues 
acknowledge the possibility that teachers and administrators could proactively consider, 
adopt, and make incremental adjustments to accommodate converging reforms. They 
warn, however, that a likely outcome would be pro forma and superficial 
implementation as opposed to deep implementation. 

An important focus of the recent literature on converging and conflicting reforms 
has been on the relationship between district-level policy and school-level reform 
initiatives. For some time, the literature has pointed to the important role that school 
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districts play in local school improvement. Districts can have a strong effect on the 
implementation and quality of school-level reform initiatives . 97 They can help local 
school reform succeed by setting the tone for local initiatives, establishing priorities 
and expectations, and allocating resources . 98 Moreover, districts can shape the 
direction for instructional reform at the school level through central programs of 
professional development, curriculum guidelines and materials, and student tests and 
assessments . 99 Indeed, studies of instructionally effective schools find substantial 
consistency and coordination between district policy and local school improvement 
goals . 100 Studies of comprehensive school reform initiatives have found that support 
and guidance as well as effective coordination from the district level are critical assets 
to the reform effort . 101 In all, there is a general consensus in the literature that district 
support is needed in order to promote and sustain change at the school level and to 
“scale-up” school-level reforms to the system level . 102 

A key issue raised in recent literature on educational reform is the potential 
conflict between local school reform efforts and accountability systems introduced at 
district and state levels. In his study of New American Schools, Thomas Glennan 
found that the lack of alignment between local school reform efforts and district and 
state accountability and testing systems significantly impeded implementation of local 
reform . 103 Likewise, in their case studies of New American Schools, Susan Bodilly and 
Mark Berends found that new methods of teaching and learning were often 
abandoned in favor of preparation for standardized tests . 104 They observed that high- 
stakes testing can work at cross purposes. It can motivate teachers and schools to 
adopt new curriculum and instructional strategies associated with local reform 
initiatives, but at the same time discourage teachers and administrators from adopting 
a richer, more in-depth curriculum. In yet other studies, comprehensive whole-school 
reforms were compromised by teachers’ fears that implementing such reforms would 
harm student performance on assessments . 105 

This study gathered numerous pieces of evidence of similar tensions and conflicts 
between CPS policy and local school improvement initiatives promoted by the 
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Challenge and its External Partners. This evidence suggests that where tensions and 
conflicts were perceived, it was usually the Annenberg work that was compromised. 

The 1997 and 2001 principals surveys asked principals to report the extent to 
which the system’s priorities conflicted with those of their Annenberg networks. 
While there were variations in their responses, 26 percent of Annenberg principals 
observed in 1997 that CPS priorities often conflicted with those of their networks 
(see Table 24). In 2001, that percentage increased to 36 percent. 



Table 24. Percentage of Annenberg Principals Who Agreed with the Statement “Central Office 
priorities often conflict with those of our network,” 1 997 and 2001 





1997 


2001 


Strongly Agree 


6% 


7% 


Agree 


20% 


29% 


Disagree 


68% 


56% 


Strongly Disagree 


5% 


8% 



Conflicts were also reported in interviews of External Partners. One-fifth of the 
30 Partners who were interviewed in 1996 cited difficulty in carrying out their 
network’s activities in the face of changing CPS policies. According to these Partners, 
the system’s elimination of student social promotion, tying promotion to 
standardized test performance, mandatory summer school for low-achieving students, 
and the threat of academic probation and reconstitution disrupted their work and 
distracted schools’ attention from long-term network goals. 106 

External Partners reported similar problems in subsequent interviews. Seven of 
the nine Partners that were studied in-depth for this project’s first report on External 
Partners reported that the CPS central administration’s emphasis on the ITBS, 
including the score-based policies of student retention and school probation, posed 
obstacles to the school improvement activities they promoted. 107 The Partners 
described several problems generated by pressure to score well on the ITBS. Teachers 
were so preoccupied with teaching directly to the test, and so many school activities 
were oriented in this direction, that they had little opportunity to engage in 
professional development on other issues, such as selecting engaging and rigorous 
children’s literature or developing a positive learning climate. In some cases, the goals 
of both Partners and school staff extended beyond teaching proficiency in basic skills 
to teaching higher order thinking, complex problem solving, and project-based 
learning. Since the standardized tests failed to assess these intellectual processes, the 
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pressure to succeed on the tests diminished the importance of these other educational 
goals and thereby undermined the efforts of Partners and schools to achieve them. 
One field research school was close to being placed on academic probation and was 
assigned a partner by the CPS central office to help it improve. The CPS partner 
encouraged the school to teach to the test while the Annenberg External Partner 
worked to help teachers implement more intellectually challenging instruction. 

External Partners identified other disruptions associated with CPS administrative 
procedures. They observed, for example, that with little advance notice, the CPS 
central administration would issue directives for principal or teacher meetings that 
disrupted or forced cancellation of Annenberg activities scheduled long in advance. 

In interviews conducted in 2001, Partners continued to report that CPS mandates 
and administrative procedures, particularly testing and high-stakes accountability, 
interfered with their goals for local school development. 108 Thus, from the beginning 
to the end of the Challenge, even the most persistent Partners felt challenged in their 
work by conflicting CPS policies and procedures. 

It could be problematic to take at face value such criticism of the CPS 
administration by Annenberg principals and External Partners. After all, the argument 
goes, both have a self-interest to appear successful and to provide alternative 
explanations for lack of accomplishment. However, evidence from longitudinal field 
research provides corroborating evidence of the tensions and conflicts reported by 
principals and External Partners. This project’s technical report on Annenberg school 
development between 1996 and 1999 presented a somewhat complex picture of the 
relationship between CPS policy initiatives and the work of the Challenge. 109 It 
described some areas where the system’s initiatives and the Challenge’s efforts to 
promote school development were compatible and mutually supportive. For 
example, at some of the field research schools, the system’s capital development 
initiative for school repairs and new school construction was instrumental in 
developing learning climates that were more conducive to teaching and learning. 

On the other hand, the report concluded that the Challenge promoted a reform 
agenda that at times collided with specific system policies, creating tensions and 
dilemmas for principals and teachers at the school and classroom levels. 
Observations, school-level documents, and interviews with school personnel other 
than principals and External Partners indicated that nowhere were the tensions and 
dilemmas between the Challenge and the system more sharply pronounced than in 
the interaction between high-stakes standardized testing and efforts to improve 
instruction. When these tensions and dilemmas were examined closely, however, it 
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was clear that high-stakes testing, coupled with the system’s probation and student 
retention policies, could play a positive and even necessary role in creating a press for 
accountability and a perceived need for change. These policies could move a school 
from complacency into action. At the same time, the evidence indicated that high- 
stakes testing could push teachers and principals to focus on the quickest means 
available to achieve administrative compliance — test preparation — and to abandon, 
or push aside at least for a while, efforts to achieve more ambitious, long-term 
instructional improvement. 

CPS student testing, retention, and school probation policies were among the 
strongest motivators for change that were documented among Annenberg schools. In 
most cases, these policies and the emphasis they placed on student performance on 
the ITBS put pressure on principals to improve test scores or risk sanctions. They 
also influenced teacher practice. In all but two of the 14 field research schools, teachers 
and other staff members expressed concern about test scores. This concern appeared to 
affect not only teacher classroom practice, but also constrained most schools’ efforts to 
develop. 

Data from the project’s field research indicated clearly that when the Challenge’s 
priorities came into conflict with CPS policy, the Challenge came in second. It is not 
clear the extent to which CPS policy and the Challenge’s efforts found a way to co- 
exist in Annenberg schools. Likewise, it is not clear the extent to which CPS policy 
may have overwhelmed or “swamped” the Challenge’s initiatives across the large 
number of schools it supported. While it is difficult to gauge the extent of the 
influence, it is clear that CPS policy, while supportive and a positive stimulus in 
some respects, served as a general countervailing force on Annenberg school 
improvement initiatives. 



Loss of Initial Improvement 

Part Two of this report identified several areas of school leadership and teacher 
professional community where until 1999, Annenberg schools seemed to be 
improving at a stronger rate than demographically similar schools that did not 
participate in the Challenge. These areas included inclusive leadership, joint problem 
solving, teacher influence in school-level decision making, and teacher commitment. 
After 1999, these initial improvements disappeared. By 2001, levels of development 
in Annenberg schools were statistically equivalent to those in non-Annenberg 
schools. 

As this discussion of how loss of initial improvement may have occurred begins, 
it is important to note that the literature on educational change has long described 
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school improvement as a long and fragile process. 110 Progress can disappear with a 
reduction in resources or the loss of key personnel; succumb to external pressures; 
and collapse under new demands that the work of change imposes on teachers, 
administrators, and school organizations. 

Several coincident changes in the implementation of the Chicago Challenge and 
in CPS policies may have had some regressive influence on the initial potentially 
promising development of Annenberg schools. Recall that between 1996 and 1999, 
the Challenge both expanded the financial resources it made available to schools and 
increased the professional support it provided to schools and External Partners. In 
1999, it reached its peak level of per-school funding. It had sponsored workshops on 
its organizational themes of time, size, and isolation. It had begun to place greater 
emphasis on teacher professional development, whole-school change, and improving 
student achievement. It had begun working more directly with its External Partners 
and had provided support and guidance to Partners and networks to develop more 
creative and effective proposals for funding. Finally, it was beginning to introduce 
workshops on improving instructional quality. At the same time, CPS was 
intensifying the high-stakes accountability policies it introduced in 1996. It had 
begun to retain greater numbers of students and raise the threshold for student 
promotion. It also raised the level of student performance on the ITBS that had to be 
achieved if schools were to avoid academic probation. 

As CPS intensified its accountability policies, the Challenge changed course. In 
1999, it began its Breakthrough School initiative, providing these specially selected 
schools continued funding in its last two years that, on average, was slightly greater 
than that year’s peak average funding levels (possible explanations for the relative 
success of the Breakthrough Schools are discussed below). The Challenge continued to 
provide support to almost 200 other schools but, by shifting a substantial portion of its 
remaining financial resources to the Breakthrough Schools, it dramatically reduced the 
amounts it provided them, from a peak of $46,983 in 1 999, to $28,808 in 2000, to 
$2,553 in 2001 . At the same time, it focused more of its direct professional support on 
the Breakthrough Schools, leaving other schools with more general professional 
support from staff. 

While the evidence is only suggestive, it is possible that the loss of Annenberg 
support coupled with the growing press of CPS accountability policies may have 
made it more difficult for teachers to participate in Annenberg activity. While there 
is no evidence to directly attribute declines in teacher participation to the loss of 
Challenge funds or to the growing demands of CPS policies, the evidence is clear 
that the frequency of teacher participation in Annenberg activities was substantially 
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lower in 2001 than in 1997 (see Table 19). The field research documented numerous 
cases where loss of Challenge funds resulted in the loss of key personnel provided by 
Partners, thereby compromising the Partners’ ability to work with their schools. Loss 
of personnel was often followed by declines in teacher commitment to and 
participation in Partner-sponsored development activity. Moreover, in interviews 
conducted throughout the research project, Partners consistently pointed to tensions 
and conflicts between their work and CPS accountability policies. Finally, the field 
research provides independent documentation of cases where pressure from CPS 
policy drew schools away from Annenberg activity and eroded progress that had been 
achieved. 



Relative Success of Breakthrough Schools 

Between 1999 and 2001, the Challenge’s 18 Breakthrough Schools achieved greater 
success in the development of teacher professional community and, to some extent, 
school leadership and relational trust than did other Annenberg schools. Recall from 
the previous section that Breakthrough Schools were successful in that they improved 
slightly or maintained initial improvements while other Annenberg schools regressed. 
As stated earlier, this research was not designed to develop evidence to explain 
differences in development between Breakthrough Schools and other Annenberg 
schools. Indeed, this initiative was not introduced until 1999, three years after the 
research was designed. Nevertheless, the research project did produce evidence that 
suggests several possible explanations for the relative success of the Breakthrough 
Schools. 

These explanations are of two sorts. The first is that Breakthrough Schools were 
different than most Annenberg schools in that they had greater capacity for 

development. The second is that unlike most Annenberg schools, Breakthrough 
Schools were able to draw on a different and perhaps a stronger set of resources to 
support their development. Each of these possibilities is explored below. 



Greater Capacity for Development 

In selecting Breakthrough Schools, Challenge staff recognized some qualities that 
allowed them to characterize these schools as more “on board” and “farther along” in 
their development than other Annenberg schools. Although analyses of survey data 
reveal no statistically significant differences in 1999 between Breakthrough Schools 
and other Annenberg schools on any indicator of the Essential Supports (with only 
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one exception), other evidence appears to corroborate the Challenge’s assessments 
and suggests that there may have been some small but important differences between 
the two groups of schools. 

Breakthrough Schools were chosen because the Challenge staff saw them as 
having greater capacity than Annenberg schools generally to make substantive 
improvement; that is, they had developed somewhat stronger capacity on which to 
develop further. While not statistically different, Breakthrough Schools as a group 
were slightly stronger on a number of key indicators of organizational capacity than 
other Annenberg schools in 1999, including teacher influence in decision making, 
teacher peer collaboration and reflective dialogue, faculty focus on and collective 
responsibility for student learning, school orientation toward innovation, and trust 
among teachers and between teachers and their principals. Together, these slight 
differences may have given Breakthrough Schools a somewhat stronger foundation 
for further improvement. It is important to note that by 2001, Breakthrough Schools 
achieved a statistically significant advantage over other Annenberg schools on these 
measures. 

In addition, Breakthrough Schools may have had an advantage of somewhat 
greater stability in their administrative leadership and faculties. As discussed above, 
school improvement may be more difficult to achieve with frequent turnover among 
personnel. The Challenge staff specifically considered the consequences of principal 
turnover when selecting Breakthrough Schools. For example, a new principal was 
hired in one of the schools initially identified to be a Breakthrough School. When 
she raised concerns about the school’s participation in the Challenge and the 
relationship with the school’s External Partner, the Challenge eliminated the school 
from its list of candidates. 

Data from the principal surveys suggest that there may have been somewhat 
greater stability in the faculties of Breakthrough Schools than in Annenberg schools 
generally. As described earlier, Annenberg principals reported hiring on average 3.6 
new teachers to their schools each year between 1995 and 1999. Principals of 
Breakthrough Schools reported hiring 3.0 new teachers on average each year during 
this period. Between 1999 and 2001, Annenberg principals reporting hiring on 
average 3.2 new teachers per year, while Breakthrough School principals reported 
hiring 2.3 new teachers per year. As cautioned earlier, the number of new teachers 
hired is not the best indicator of teacher attrition. Nevertheless, it is one indicator of 
“churn” among personnel that may challenge a school’s ability to introduce, develop, 
and sustain improvement over time. And, while this indicator points to substantial 
“churn” among faculty in Breakthrough Schools, it was less than that experienced by 
other Annenberg schools and warrants consideration. CPS personnel records for 
1997-98 confirm such a difference, indicating that 13 percent of teachers in 
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Breakthrough schools were new to their schools, whereas about 16 percent of teachers 
in other Annenberg schools were new to their schools that year. 

Schools were also selected for Breakthrough status and funding because of their 
relatively greater commitments to the Challenge. One of the primary criteria used by 
Challenge staff in Breakthrough School selection was participation in Challenge- 
sponsored activities. On average, these schools were perceived by Challenge staff as 
having made more effective use of Challenge resources and had participated more 
frequently in Challenge activities. If the Challenge staff was correct in its assessments, 
these differences may account in part for some advantages Breakthrough Schools 
seem to have had in promoting school improvement. 

As discussed earlier, another indicator of school commitment is teacher 
participation in Annenberg activities. Principals survey data reveal substantial 
differences between Breakthrough Schools and other Annenberg schools on this 
score. As shown in Table 25, 38 percent of Breakthrough School principals reported 
in 1997 that 80 percent or more of their teachers participated regularly in Annenberg 
activity whereas only 1 5 percent of principals of other Annenberg schools reported 
such high levels of participation. Although this difference narrowed somewhat in 
1999, in 2001 almost twice the percentage of Breakthrough principals reported this 
high-level teacher participation. It is also important to note that in 2001, no 
principal of a Breakthrough School reported that fewer than 40 percent of her 
teachers participated in Annenberg activities. This stands in contrast to the finding 
that one-third of all Annenberg principals reported that year that fewer than 40 
percent of their teachers participated in Annenberg activity. These relatively higher 
rates of teacher participation, coupled with relatively lower indicators of teacher 
turnover point to a potential advantage among Breakthrough Schools of having 
somewhat more stable, critical masses of teachers to promote school improvement. 



Table 25. Percentage of Principals of Breakthrough and Other Annenberg Schools Reporting That 80 
Percent or More of Their Teachers Participated in Annenberg Activities, 1997 to 2001 





BREAKTHROUGH SCHOOLS 


OTHER ANNENBERG SCHOOLS 


1997 


38% 


15% 


1999 


40% 


32% 


2001 


44% 


23% 



Different and Sustained Resources 

The Breakthrough School initiative provided more concentrated resources to a group 
of schools that may have had, on average, a stronger capacity for development. It was 
noted earlier that on average Breakthrough Schools received sustained and somewhat 
higher levels of funding during the Challenge’s last two years while funds provided to 
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other Annenberg schools were substantially reduced (see Figure 5). In 2000 and 
2001, Breakthrough Schools received a yearly average of nearly $50,000 while other 
Annenberg schools received about $2,600. As argued earlier, while $50,000 a year 
may not purchase very much, sustained support even at this modest level might have 
helped Breakthrough Schools maintain their relationships with their External 
Partners and sustain their improvement efforts. Moreover, Breakthrough Schools 
received these funds directly from the Challenge rather than through their Partners, 
giving them potentially more money and more discretion in its use. In addition to 
sustained levels of funding, Breakthrough Schools may also have benefited from 
ongoing professional support from Challenge staff. 

Breakthrough Schools may also have had some advantages over other Annenberg 
schools with regard to their External Partners. On one hand, the types of Partners that 
worked with Breakthrough Schools were roughly similar to those that worked with 
Annenberg schools as a whole, although a somewhat greater proportion of 
Breakthrough School Partners were university based (see Table 26). Breakthrough 
Schools belonged to networks with about the same number of schools on average as 
other Annenberg networks so they did not have fewer schools with which to compete 
for their Partners’ attention and resources. On the other hand, Breakthrough Schools 
were somewhat more likely than other Annenberg schools to have had Partners with 
experience working in schools prior to the Challenge. About three-quarters of Partners 
working with Breakthrough Schools had worked previously with schools on long-term 
improvement projects. Overall, two-thirds of Partners working with Annenberg 
schools had such experience. This difference suggests that as a group Breakthrough 
Schools may have had Partners with stronger expertise for promoting school 
development than Annenberg schools generally. In addition, two-thirds of 
Breakthrough Schools had worked with their External Partners prior to the 
Challenge. Breakthrough Schools and their Partners may have had more well- 
developed working relationships overall than other Annenberg schools. This creates 
the possibility that Breakthrough Schools may have had a relatively stronger base of 
social resources through their relationships with their Partners, including relational 
trust and mutual accountability, that helped promote school improvement. 
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Table 26. Percentage of Types of Annenberg External Partners Working with Breakthrough Schools 
and All Annenberg Schools 



TYPE OF PARTNER 


PERCENT OF BREAKTHROUGH 
SCHOOLS (N*ll) 


PERCENT OF ALL ANNENBERG 
SCHOOLS (N=43) 


University 


45% 


35% 


Educational Services 
Organizations 


27% 


28% 


Community Organizations 


9% 


14% 


Cultural Institutions 


18% 


23% 



There is also some indication that Breakthrough Schools may have been better 
able than Annenberg schools generally to parlay their participation in the Challenge 
into additional resources. Data from the 2001 principal survey indicate that 
Breakthrough School principals were somewhat more satisfied with the benefits of 
their participation in the Challenge than Annenberg school principals. That year, all 
Breakthrough School principals agreed or strongly agreed that participation provided 
their schools with useful resources. Ninety percent of all Annenberg principals made 
similar assessments. All of the Breakthrough School principals agreed or strongly 
agreed that participation in the Challenge provided resources or in-kind services 
needed for improvement. This compares to 84 percent of all Annenberg principals 
who made similar assessments. Finally, in the most substantial point of contrast, 
whereas all Breakthrough School principals agreed or strongly agreed that 
participation in the Challenge strengthened their schools’ ability to obtain resources 
beyond those that the Challenge provided, only 69 percent of all Annenberg 
principals so agreed. 

In addition, Breakthrough Schools were somewhat more likely than Annenberg 
schools to be members of networks with more comprehensive and therefore potentially 
more effective school improvement foci (see Table 27). As argued earlier, efforts 
emphasizing the coordinated development of mutually influential Essential Supports 
may be more effective than those focusing on only one Support to the exclusion of 
others. A smaller proportion of Breakthrough Schools were in networks focusing 
primarily on curricular and instructional improvement, suggesting perhaps that greater 
proportions of Breakthrough Schools were working on curricular and instructional 
improvement in more comprehensive and potentially more effective ways. 
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Table 27. Percentages of Breakthrough Schools and All Annenberg Schools by 
Primary Network Focus 



PRIMARY NETWORK FOCUS 


PERCENT OF 
BREAKTHROUGH 
SCHOOLS 


PERCENT OF ALL 
ANNENBERG SCHOOLS 


Curricular and Instructional Improvement 


39% 


51% 


Learning Climate and Social Services 


17% 


20% 


Partner and Community Development 


11% 


10% 


“Comprehensive” 


33% 


19% 



In sum, a number of factors might explain the relative success of Breakthrough 
Schools. The evidence suggests that Breakthrough Schools may have had somewhat 
greater capacity for development than Annenberg schools generally. They seemed to 
have lower turnover in administrative leadership and faculty. Breakthrough Schools 
had generally higher levels of teacher participation in school-level Challenge activities 
and as a group they participated more regularly and consistently in Challenge- 
sponsored programs. Their Challenge funding was sustained at a slightly higher level 
over a longer period of time. Moreover, they received their funds directly from the 
Challenge rather than through their External Partners, giving them potentially more 
money with which to work and more discretion in its use. Breakthrough Schools 
received more sustained professional support from Challenge staff. In addition, they 
may have had access to different resources through their External Partners. As a 
group, greater proportions of their Partners had previous experience working in 
schools and they were more likely to have had working relationships with their 
Partners that extended to before the Challenge. Breakthrough Schools may have been 
better able to leverage their participation in the Challenge and their relationships 
with their Partners to obtain additional resources. Finally, greater proportions of 
Breakthrough Schools belonged to networks with more comprehensive and 
potentially more effective foci for promoting school development. It is likely that no 
one of these factors explains the relative success of Breakthrough Schools 
documented in this report but that a number of factors worked in combination to 
promote improvement. 

One final comment is in order. The relative success of Breakthrough Schools may 
also be attributed to a motivational boost that may have accompanied the award of 
Breakthrough status and funding. According to one of Annenberg’s External 
Partners, 

I think what this served to do, and you see this in a lot of areas, is the schools [that] 
got the Breakthrough grants felt motivated and honored, and [this made them 
think] “We’re not going to let them down, we’re going to do it.” And the other 
[schools] are kind of like, “Well, we didn’t get that money.” 
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How much of a motivational boost this initiative provided is unclear. Neither is how 
much influence such a boost might have had on the ongoing development of 
Breakthrough Schools. It may have played an important role in sustaining 
commitments to promote school improvement at a time when support in other 
Annenberg schools was waning and the influence of CPS policies was intensifying. It 
remains to be seen whether the Breakthrough Schools sustained their progress after 
the Challenge shut its doors and the financial support, professional support, and 
motivational impetus it provided had ended. 



Summary 

This section addressed the third general question of this research: What factors might 
explain the improvement or the lack thereof among Annenberg schools? A number of 
factors were discussed that provide possible explanations for the lack of overall effect of 
the Chicago Challenge on school improvement, the loss of initial improvement 
midway through the Challenge, and the relative success of the Breakthrough Schools. 

In summary, the failure of the Challenge to achieve an overall effect on school 
improvement could be due to a number of shortcomings in the design and 
implementation of the Challenge itself. These include the breadth of its goals and the 
vagueness of its strategies for school development; the numbers of participating 
schools and the inadequacy of the resources they received; and general weaknesses in 
the levers for change that it developed, particularly with regards to accountability. 
Failure to achieve an overall effect could also be due to weakness in the capabilities and 
resources of the External Partners and the organizational capacities of Annenberg 
schools to engage effectively in the Challenge's approach to reform. Finally, CPS 
policies could have acted as a countervailing force at the school level. Most likely, no 
one of these factors alone would fully explain the lack of an Annenberg effect. Rather, 
these and perhaps other factors worked in combination. 

The loss of initial improvement among Annenberg schools in some areas of the 
Essential Supports could be explained by a convergence of intensifying CPS 
accountability policies with the Challenge's shift in strategy to focus its efforts on 
Breakthrough Schools and reduce its support of others. The evidence suggests that 
reductions in support occurred concurrently with teachers' growing concern about 
CPS accountability policies, declining teacher participation in Annenberg activity, 
and an increase in the difficulty External Partners faced in sustaining their work with 
less funding from the Challenge. 

Finally, the relative success of Breakthrough Schools could be explained by their 
somewhat greater capacity for improvement and for engaging in Annenberg-style 
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reform. When they were selected in 1999, these schools had somewhat stronger 
leadership and professional community than other Annenberg schools. They had 
somewhat greater stability in their teaching and administrative staffs. In addition, 
they exhibited stronger commitments to the Challenge in their histories of 
participation in Challenge-sponsored activities and teacher participation in school- 
level Annenberg work. Finally, Breakthrough Schools had access to different 
resources than other Annenberg schools. They received sustained financial support 
from the Challenge for two additional years while funds for other Annenberg schools 
were withdrawn. Finally, the networks and the External Partners with which they 
were associated may have had qualities that distinguished them from networks and 
Partners generally associated with other Annenberg schools, including longer 
working relationships on which to build, stronger experience among Partners in 
working with schools, potentially stronger Partner expertise in school improvement, 
and more comprehensive foci on school improvement. 
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The experience of the Chicago Annenberg Challenge suggests a number 
of lessons about promoting large-scale school improvement, or the 
simultaneous improvement of many schools in different contexts. Several 
of these lessons are presented below as answers to particular questions 
that arise when developing and implementing such initiatives. These 
lessons relate in a number of ways to the field research findings presented 
in Part Two about what makes individual school improvement 
successful. Such relationships are noted as relevant. 

How to Promote Large-scale School Improvement: 

One Way? Any Way? or Better Ways? 

The literature on educational change makes clear that there is no one best way to 
improve individual schools or groups of schools. As Richard Elmore writes, 
“[Improvement is] a function of learning to do the right thing in the setting where 
you work .” 112 Numerous studies have found that successful school improvement 
requires the discretion of local actors to identify and solve site-specific problems and to 
adapt programs and policies to meet local needs . 113 Indeed, some studies contend 
that local self-determinism is essential to build the commitments necessary to 
implement and institutionalize reform into the life of the school. 

At the same time, the literature also argues that some strategies for improvement 
are better than others. In other words, some can supply Elmore’s “right thing” in a 
more efficient and effective way. Fullan and Miles contend that one of the main 
reasons educational reforms fail is that they are often based on “faulty maps of 



112 Elmore (2000). 

113 See Fullan (2001) and McLaughlin (1990) 
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change .” 114 Maps of change refer to assumptions about how change happens, the 
means required to achieve specific ends, and perhaps assumptions about conditions 
that must be in place for those means to operate effectively. Fullan and Miles argue 
that some of the maps for reform initiatives are too vague to provide reliable or valid 
guidance. Some fail to recognize the complexities of schools and the broader system 
of schooling. Some are directly contradicted by empirical evidence. Others, while 
attractive politically, do not work and may even create new problems or exacerbate 
the problems they were intended to solve. Fullan and Miles argue that important 
change cannot be mandated. Instead, change requires “skill, commitment, 
motivation, and discretionary judgment on the part of those who must change .” 115 
And yet, even though local discretion is important to successful change, they contend 
that consistently reliable and effective maps can and should be used to guide 
improvement within and across schools. 

The experience of the Chicago Annenberg Challenge illustrates a problem that 
reformers face when trying to figure out how best to promote improvement among a 
large number of schools. Following the national Annenberg Challenge, the Chicago 
Challenge was founded on the well-established premise that there is no one best way 
to promote local school development. Adding to this premise a view of the 
importance of local initiative and control in school development and faith in 
decentralization and democratic localism, the Challenge eschewed common goals and 
specific processes. It sought to guide local development in particular directions and to 
provide some measure of accountability to focus schools’ efforts in these directions. 
Moreover, it laid out reform principles of pluralism and local self-determinism. What 
the Challenge did not do, however, was “privilege” one reform strategy over another. 
It left local school communities to set their own goals and strategies for development. 
As a result, it ended up supporting a wide range of local strategies, with some no 
doubt less well informed and less effective than others. 

Both the Challenge’s experience and the literature on educational change point 
to a middle ground. As Fullan and Miles suggest, between “one way” and “any way” 
are “better ways” to promote improvement among groups of schools. While it may 
be important to encourage local pluralism and self-determinism in developing, 

. adopting, and implementing initiatives to make schools better, it may be equally 
important to provide guidance for local initiatives in the form of well-researched and 
well-thought-out maps for change. Such maps would not impose scripts for local 
actors to follow; rather, they would present sound theories and principles that might 
enhance the effectiveness of local thinking and action. Some insights into what such 



114 See Fullan and Miles (1992) and Argyris and Schon (1975). 
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theories and principles might be were presented at the end of Part Two in the 
discussion of what makes local school improvement successful. 



Which Schools to Support? 

This study raises the important issue of which schools should be supported and in 
what kinds of reform they should engage. As argued in Part Three, different 
reforms make different kinds of demands on schools and the success of any 
particular reform may depend on the capacity of schools to engage in and 
implement that reform well. The implication is that if a school or a group of 
schools lacks the capacity to implement a particular reform well, another type may 
be warranted. 

As discussed in Part One, the national Annenberg Challenge made a clear 
argument that local school reform is best pursued through a plurality of approaches 
that privileges none. The Chicago Challenge was designed on this principle. 
Proceeding from this principle, however, both the national Annenberg Challenge and 
the Chicago Challenge paradoxically promoted one particular approach — one that 
was local, collaborative, and self-directed. Evidence in Part Three showed that 
substantial proportions of the schools receiving support from Chicago Challenge 
were weak in key organizational capacities of leadership and professional community, 
that arguably would be important to implement this type of reform. Indeed, it is 
unlikely that schools that were particularly weak in these and perhaps other 
organizational capacities would be able to take full advantage of the opportunities 
that participation in the Challenge extended . 116 And as argued in Part Three, this 
might be one of the reasons for the Challenge's failure to achieve an overall effect on 
school development. Other reform strategies might have been more appropriate and 
more effective for these schools. Such strategies might have included efforts to hire 
and retool faculty and administrators, new systems of accountability, and more direct 
intervention from the CPS central administration or external organizations. 

On the other hand, there is evidence that one of the reasons for the relative 
success of the Breakthrough Schools is that they were chosen, in part, because they 
possessed somewhat stronger capacities to “do Annenberg.” One can make the 
argument that the relative success of the Breakthrough Schools was due to a selection 
bias, that the Challenge hand-picked schools with particular qualities that gave them 
an “edge” to succeed, and rightly so. The Breakthrough School initiative represented 



116 See Hargreaves (2003). 
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a significant departure from the earlier, less discriminating, and perhaps less effective, 
strategy that the Chicago Challenge used to identify schools to support. 

The experience of the Chicago Challenge also raised the important issue of how 
many schools a large-scale reform effort ought to support. How this issue is addressed 
depends on a number of factors, including the amount of resources that are available, 
the ability of those responsible for the reform to manage those resources well, and, 
taking the point above, the number of schools that may have the capacity to 
implement the reform well. The Chicago Challenge used two different strategies to 
address this issue. Between 1995 and 1998, it spread its resources thinly among as 
many as 21 1 schools, or nearly 40 percent of all Chicago public schools, through up 
to 45 networks and External Partners. Moreover, it took upon itself the substantial 
burden of providing some measure of professional support to all the schools and their 
Partners. 117 In 1999, it changed course and redirected its remaining resources to a 
smaller number of selected schools. As one Challenge staff member explained in an 
interview: 



It was a different way of doing business than we had done before. I 
guess it was just our effort to say whole-school change requires more 
resources than what we had initially... not what we had initially 
thought. 

While not conclusive, the relative success of the Breakthrough Schools suggests that 
it is more effective to concentrate greater amounts of resources on a smaller number 
of schools that are selected in part for their capacity to implement the reform well. It 
is less effective to distribute relatively small amounts of resources among a very large 
number of schools that have been selected with less discrimination. 



What Resources Are Needed? 

The literature on school change and this research on the Chicago Challenge 
indicate that resources matter a great deal in the promotion of improvement among 
individual schools or groups of schools, particularly among those that are 
underresourced. This study suggests that financial resources are important to school 
improvement; that the provision of stable financial support over time may be 
associated with ongoing improvement and that the loss of resources, particularly 
early on in the reform, may slow or terminate improvement. Recall that most 
Annenberg schools received two or maybe three years of “full” support before 1999, 



117 The processes by which the Challenge made its early grants are described in detail in Shipps and 
Sconzert with Swyers (1999). 
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at which point the Challenge reduced substantially its general financial support. As 
will be discussed later in this section, two to three years is not a lot of time for the 
hard, steady work required to improve schools. 

It is important to note that the particular resources that are needed to promote 
school improvement are likely to depend on a number of considerations. The type 
and amount of the resources are likely to be contingent upon what resources the 
system already has and what resources are at the school’s command. Schools and 
school systems that are poorly funded may need substantially more financial support 
than those that have more money at their disposal. Schools that are weak in 
organizational and human capacity may need additional personnel support and 
central guidance. The type and amount of resources would also depend on the 
ambitiousness of the improvements that are sought. The assumption is that the more 
ambitious the improvement, the more resources are required. 

That said, this research provides little guidance as to how much money may be 
needed to promote lasting improvement in individual schools or among a large 
number of schools in an underresourced urban system like Chicago’s. The highest 
level of average per-school funding that the Challenge provided was only about 1 percent 
of an elementary school’s annual operating budget. Certainly, this money was 
helpful — Annenberg principals and External Partners said as much on surveys and in 
interviews. Still, in underresourced urban schools, the average amount of money the 
Challenge allocated provided very little support relative to the likely need. As the 
Breakthrough School initiative suggests, the Challenge might have had more success 
allocating larger amounts of resources to a smaller number of schools with greater 
capacity to engage in its particular approach to reform. 

The findings presented here suggest that while financial resources are important 
to the improvement of individual schools and groups of schools, how the money is 
spent matters more. It was beyond the scope of this work to engage in an in-depth 
study of network and school-level budgets. Field research and interviews with 
External Partners reveal that Annenberg funds were used to support a wide range of 
activities, some of which helped promote school improvement and some of which 
did not. The field research documented schools that were rich in accumulated 
resources but made little productive use of them. It also documented schools that 
strategically acquired and allocated their funds to align with and support their school 
improvement goals and activities. As shown in comparisons of improving and 
nonimproving schools in Part Two, it was the strategic acquisition, allocation, and 
alignment of resources rather than mere acquisition that seemed to be associated with 
individual school improvement. 
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Another insight from this research is that money appears to be a necessary but 
insufficient resource to promote and support the improvement of individual schools 
and groups of schools. The nature of the external support that is provided is also 
important. This study identified several sources of external professional support that 
might be important to school improvement, from External Partners, to relationships 
among teachers and principals at different schools, to the Challenge itself as the 
central sponsoring agent of development. Findings pointed to the importance of 
external experience and expertise in developing strong “theories” of change and 
effective school improvement strategies. They pointed to the importance of social 
capital — the resources of trust, shared expectations, and mutual accountability — that 
come from strong working relationships. The findings also suggested that political 
capital was important to help buffer schools from conflicting external influences and 
to link them with still other resources to promote improvement. As was the case with 
the Breakthrough Schools, it seems to be a combination of strong and varied 
resources that are sustained for some period of time that matters most to school 
improvement. 

Finally, this study points to the importance of alignment or coherence among 
resources for successful school improvement. The findings revealed difficulties that 
school personnel and Partners faced as they tried to promote improvement in schools 
with multiple, conflicting programs and reform initiatives. Problems occurred when 
local school improvement initiatives and the work of the Challenge itself as a large-scale 
initiative conflicted with the school system’s policies. As an earlier report on instructional 
program coherence demonstrated, and as the cases of improving and nonimproving 
schools in Pan Two illustrate, school improvement, be it in individual schools or among 
a large group of schools, appears to be enhanced when resources and action cohere 
around a shared agenda. 118 When resources and action are fragmented and pull schools 
in multiple and perhaps conflicting directions, improvement is less likely to occur. 



Working With or Against the System ? 

The Chicago Challenge promoted local school reform, but it also had an agenda to 
change the school system (a matter that was not explored in the research). As such, it 
was designed to be “in the system” but not “of the system.” It was set up to work 
against the bureaucracy and centralized policies and practices that were believed to be 
constraining local school governance and improvement. However cooperative the 
Challenge’s initial relationship with the CPS central administration was when the 
Challenge was founded, the 1995 reform changed everything. As discussed in Part One, 



118 Newmann et al., (2001b). 
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both structurally and politically, the Challenge had difficulty developing a productive 
working relationship with the system’s central administration after 1995- While its 
leadership sought to cultivate a working relationship with CPS leadership, it never 
achieved a level of cooperation that might have been conducive to its efforts to promote 
local school improvement. 

The experience of the Chicago Challenge raises a dilemma in thinking about the 
relationship between large-scale reform initiatives and the school systems in which 
they operate. On one hand, the critical perspective of central system bureaucracy that 
the founders of the Challenge held had substantial merit. The failures of the CPS 
central administration and its lack of accountability were legion; they were primary 
reasons for adoption of both the 1988 and 1995 reforms. 119 So, there was a strong 
argument to be made that the system and its central administration were legitimate 
targets for reform. At the same time, lessons from experience and numerous studies 
of other reform initiatives conclude that efforts to improve both individual and large 
groups of schools are unlikely to be successful, at least for very long, without the 
school system’s support. In his reflections on 40 years of research on school reform, 
Matthew Miles observed that large-scale reform initiatives require continued close 
central-local interaction. 120 Local changes need to be embedded in stable and 
supportive system-level routines and linked well to system policies. 

Reform advocates face the dilemma of how to be partners with a system in order 
to support improvement across a large number of schools and, at the same time, 
confront and challenge the system itself to change. It may be extraordinarily difficult 
to manage this dilemma, but a minimal condition for success seems to be 
constructive interaction between reform and system leadership and a direct 
engagement of the dilemma. In the case of the Chicago Challenge, there was neither 
a history of constructive interaction nor the engagement of the issues related to the 
relationship between the reforms supported by the Challenge and those advanced by 
the school system. As a result, the conflicts and contradictions between the two were 
played out in the schools, often to the detriment of improvement efforts supported 
by the Challenge. 



When Are We Going to Get There f 

An additional issue raised by the Chicago Annenberg experience concerns the 
amount of time that may be required to promote and sustain school improvement. 



119 For example, see Hess (1991) and Shipps, Kahne, and Smylie (1999). 

120 Miles (1993). 
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The literature on educational change is replete with warnings that reform involves 
long, steady work. 121 It is a slow process. 122 Research on the implementation of 
comprehensive reform models reports that it can take years before teachers 
understand what a new reform fully entails. 123 Researchers have given various 
estimates of the amount of time required for schools to fully implement and 
institutionalize different types of reform. For example, Henry Levin estimated that 
it takes approximately six years for a school to transform completely into an 
Accelerated School. 124 Nancy Haynes concluded that it can take five to seven years 
to institutionalize the Comer School Development model. 125 In their study of the 
development of New American Schools, Susan Bodilly and Mark Berends found 
that even after three years, many New American Schools 5 designs were only 
partially implemented. 126 Michael Fullan contends that it takes at least three years 
to turn around a poorly performing elementary school and six years to turn around 
a poorly performing high school. 127 Linda Darling-Hammond and Theodore Sizer 
have estimated that it can easily take 10 years to completely reform a single 
school. 128 

The literature on school change indicates that the implementation of new 
reforms can be undermined if support for them is withdrawn prematurely. 129 From 
their study of “theory-based reforms, 55 Milbrey McLaughlin and Dana Mitra wrote 
that the sustainability of these efforts depends not only on an ongoing, adequate base 
of resources, but on several other conditions as well. 130 These include what reform 
advocates learn from taking a reform idea and putting it into practice and what actions 
they may take to adapt the reform to the specific conditions in which it is to be 
implemented. They also cite as necessary a thorough understanding among school 
personnel of the reform’s underlying principles, the support of the community of 
practice within the school, a knowledgeable and supportive principal, and a 
supportive district context. Developing these conditions may take a substantial 
amount of time and effort. 

In 1999, the Challenge began its Breakthrough School initiative and, at the same 
time, reduced the amount of resources it provided to other Annenberg schools. As of 



121 Elmore and McLaughlin (1997). 

122 Cohen (1994) and Cuban (1984). 

123 Bodilly (1998). 

124 Levin (1991) and Murphy and Datnow (2003). 

125 Haynes (1998). 

126 Bodilly and Berends (1999). 

127 Fullan (2001). 

128 Darling-Hammond (1990) and Sizer (1992). 

129 Bodilly (1996); Muncey and McQuillan (1996). 
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1999, most Annenberg schools had received only two or three years of support from 
the Challenge. While there was evidence that Annenberg schools as a group were 
beginning to develop in some areas of the Essential Supports at a rate greater than 
non-Annenberg schools, there were also doubts among members of the Challenge 
staff and its Board of Directors that continuing this course of action — supporting a 
large number of schools that implemented a wide variety of local initiatives, some 
better than others — would result in much overall success. These doubts fueled the 
Breakthrough School initiative. It is not clear that the initial improvement among 
Annenberg schools would have grown had the Challenge stayed its initial course. By 
most estimates in the literature, it would be unreasonable to expect to see much 
change in only two or three years. What the data from this study show, however, is 
that the reduction of support for non-Breakthrough Annenberg schools coincided 
with a loss of these initial improvements. The evidence also shows that Breakthrough 
Schools, who were provided sustained support for a total of four or five years, were 
able to build upon initial improvements and achieved greater overall success in some 
areas. 

It is easy to become impatient with efforts to improve both individual schools 
and large groups of schools. It is not uncommon to set unreasonable goals and 
unreasonable timelines to achieve those goals. It is commonplace to abandon reform 
initiatives before enough time has passed for them to take hold and succeed or fail. It is 
also commonplace to move from one reform to another without taking enough time to 
study and learn from them. 131 While it may be foolish to spend too much time and too 
many resources on bad reform strategies, it is also foolish to give up prematurely on 
potentially effective ones. There remains a great deal to learn about promoting large- 
scale school improvement, particularly in underresourced urban public school 
systems like Chicago’s. While the Chicago Annenberg Challenge did not achieve 
widespread improvement in the schools it supported, its experience leaves a legacy of 
important lessons that may guide future initiatives toward more productive strategies 
and away from less productive ones. 



131 See Slavin (1989). 
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Appendix A 



Chicago Annenberg External Partners and the 
Numbers of Schools in Their Networks 



External Partners 


No. of Schools 


Academic Development Institute 


3 


Association of Illinois Middle Level Schools 


3 


Beverly Area Planning Association 


6 


Chicago Children’s Museum 


3 


Chicago Metropolitan History Education Center 


4 


Chicago State University 


8 a 


Chicago Symphony Orchestra 


3 


Chicago Teachers Union — Quest Center 


3 


Coalition for Improved Education in South Shore 


9 


Coalition of Essential Schools Regional Center at Chicago 


6 


Columbia College — Science Institute 


3 


DePaul University School of Education 


4 


Designs for Change 


5 


Erickson Institute 


3 


Facing History and Ourselves 


3 


Garfield Park Conservatory Alliance 


4 


Governors State University 


3 


Great Books Foundation 


4 


Hug-A-Book 


3 


Illinois Future Problem Solving 


5 


Illinois Learning Partnership 


3 


Illinois Resource Center 


3 


Imagine Chicago 


4 


Kohl Children’s Museum 


3 


Logan Square Neighborhood Association 


5 


Loyola University 


4 


National Louis University — Center for City Schools 


4 


National Louis University — Faculty 


6 


Near Northwest Neighborhood Association 


5 


Northeastern Illinois University — Chicago Teachers Center 
(Group A) 


3 


Northeastern Illinois University — Chicago Teachers Center 
(Group B) 


3 
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Chicago Annenberg External Partners and the 
Numbers of Schools in Their Networks (continued) 



External Partners 


No. of Schools 


Northeastern Illinois University — Chicago Teachers Center 
(Group C) 


3 


Northeastern Illinois University — Chicago Teachers Center 
(Group D) 


3 


Northeastern Illinois University — Chicago Teachers Center 
(Group E) 


4 


North Lawndale Learning Community 


9 


Participation Associates 


3 


People’s Reinvestment Development Effort 


3 


Roosevelt University 


3 


Success for All Foundation 


3 


Suzuki-Orff School for Young Musicians 


4 


Teachers Task Force 


3 


University of Chicago — Center for School Improvement 


8 


University of Illinois at Chicago — Small Schools Workshop 


15 b 


Whirlwind Performance Company 


3 


Youth Guidance 


12 



Source: Chicago Annenberg Challenge. This list contains External Partners of networks receiving 
implementation grants in 1999. All but two of these partners continued to receive support through 
2001, the last year of the Challenge. 3 These eight schools are schools within four larger schools. b These 
15 schools include some independent small schools as well as small schools within nine larger schools. 



Appendix B 



Indicators of High and Low States of Development on the 
Model of Essential Supports for Student Learning 



Essential Support 


Low State 


High State 


High Quality 
Instruction 


• Curriculum characterized by 
slow pacing and a great deal 
of review and repetition. 


• Curriculum is well-paced 
and coordinated across 
classrooms and grade levels. 


• 


• Instruction is aimed only at 
mastery of basic skills. 


• Instruction is aimed at 
student mastery of 
challenging intellectual work 
and basic skills. 




• High quality instructional 
materials are not available or 
not used. 


• High quality instructional 
materials are used. 




• There are many disruptions 
to instruction. 


• Instructional time is 

protected form interruption. 


S tuden t-Cen tered 
Learning Climate 


• School is disorderly with 
many disruptions. 


• School is orderly. 




• Students feel 

physical/psychological risk or 
danger. 


• Students feel physically and 
psychologically safe. 




• Impersonality and alienation 
characterize teacher-student 
relations. 


• Personalism and belonging 
characterize teacher-student 
relations. 




• Teachers hold low academic 
expectations for students. 


• Teachers hold high academic 
expectations for students. 




• Students find their peers give 
them little support for 
academic learning. 


• Students find high peer 
support for academic 
learning. 



158 




174 



APPENDIX B 



159 



Essential Support 


Low State 


High State 


School Leadership 


• Principal is exclusive leader. 


• Leadership is broad based 
and includes principal, 
teachers, others. 




• Decision making is 
authoritative. 


• Decision making is 
democratic and shared. 




• Teachers to not meet 
regularly to plan 
improvements. 


• Teachers work to plan 
improvements regularly. 




• Leadership does not work to 
protect school from 
disruptive influences. 


• Leadership buffers school 
from disruptions. 




• Principal fails to articulate, 
communicate plans and 
goals of organization. 


• Principal articulates, 
communicates plans and 
goals of organization. 




• Leadership lacks focus or 
focus is not on instruction. 


• Leadership focuses on 
instruction. 




• Lack of accountability is the 
norm. 


• Principal and teachers take 
responsibility. 




• Principal fails to help 

teachers obtain professional 
development. 


• Principal promotes the 
development of teachers. 




• The school is poorly 
managed and chaotic. 


• The school is efficiently 
managed and runs on 
schedule. 


Teacher Professional 
Community 


• Teachers’ vision and goals 
are ambiguous or not shared. 


• Teachers share a clear vision 
and goals. 




• Teachers are unable to 

articulate their goals and lack 
a common language. 


• Teachers use a common 
language to articulate their 
vision and goals. 




• Social groups are fragmented 
subcultures at the school. 


• There is normative 

coherence among social 
groups and subcultures at 
the school. 




• Teachers are isolated from 
each other and do not share 
reflective dialogue, inquiry, 
or joint work. 


• Teachers collaborate through 
reflective dialogue, inquiry, 
and joint work. 
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Essential Support 


Low State 


High State 


Teacher Professional 

Community 

(continued) 


• Teachers feel responsibility 
and accountability only to 
themselves. 


• Teachers feel that they have 
a shared responsibility and 
accountability. 




• Teachers have limited 
communication channels. 


• Teachers have expansive 
communication channels. 




• There are limited structures 
and time for collaboration. 


• There are sufficient 
structures and time for 
collaboration. 




• There are disruptive, 

counterproductive political 
and intellectual tensions. 


• There are productive 
political and intellectual 
tensions. 


Parent and 

Community 

Involvement 


• Students lack parent support 
for learning at home. 


• Parents strongly support 
student learning at home. 




• The principal fails to draw 
on community resources and' 
institutions for school. 


• The principal actively draws 
on community resources and 
institutions for school. 




• School conducts little 
outreach to parents as 
resources. 


• School actively reaches out 
to parents as resources. 


Relational Trust 


• Teachers and principal feel 
distrust, cynicism. 


• Teachers and principal feel 
trust, optimism. 




• Teachers feel distrust and 
cynicism toward each other. 


• Teachers feel trust and 

optimism toward each other. 




• Teachers and parents feel 
distrust, cynicism. 


• Teachers and parents feel 
trust and optimism. 




• Teachers and students feel 
distrust, cynicism. 


• Teachers and students feel 
trust and optimism. 


School Instructional 
Program Coherence 


• Instructional programs have 
different and sometimes 
divergent goals. 


• Instructional programs share 
common focus. 




• There are so many programs 
that teachers cannot keep 
track of them. 


• There is a small enough 
number of programs that 
teachers can keep track of 
them. 
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Appendix C 

Longitudinal Field Research Methods 

In this appendix we describe in detail the procedures we used to select our school 
field research sites, our data collection procedures, and our methods of analysis. 

Selection of Sites 

In 1996 and 1997, more than 40 networks of schools and External Partners were 
awarded multi-year implementation grants by the Chicago Annenberg Challenge. 
These networks included between 200 and 220 elementary, middle, and high 
schools, approximately 90 percent of which were elementary schools. From these 
networks and schools, we selected an initial sample of 1 1 networks and 23 field 
research schools. As we described in Part One, sample selection began with the 
networks. We selected networks with diverse organizational foci, networks with both 
newly formed and well-established relationships with schools, and networks with 
different types of External Partners (e.g., universities, community organizations, and 
cultural institutions). We then selected two or three schools as research sites from 
each of these networks. One to two schools were chosen because of their promise for 
working well with their External Partners and succeeding in their efforts to develop. 
An additional school was chosen because of indications that it might struggle to 
succeed. Our intention was to create a purposive sample of schools that would allow 
us to understand reasons for more or less successful development. Our site selections 
were informed by Consortium survey data and assessments from the External 
Partners of the networks we sampled. 

We selected our sample of networks and schools in two stages. A first group 
was selected in the fall of 1996 from the networks and schools that received the first 
round of Annenberg funding. A second group was selected in the fall of 1997 from 
those receiving funding in the second round. In all, our sample included 18 
elementary and middle schools and five high schools. By the end of the 2000-01 
school year, the end point of analysis for this report, we collected five years of field 
research data from about half of the networks and schools in our sample; we collected 
three years of field research data in the other half. 

In this report we focus particular attention on 12 elementary schools. We 
chose not to focus on high schools for two reasons. First, high schools represented 
only 10 percent of schools supported by the Challenge. Second, our high school data 
were not as comprehensive as our elementary school data. We also did not include six 
of the 18 elementary/middle school sites in our analyses for this report. These schools 
either dropped out of the study, were dropped from the study because of lack of 
improvement activity, or did not participate fully in our field research and their data 
were not as complete as other schools. We chose to focus on those schools with the 
most complete evidence available. 
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Although we did not intend to select a group of schools that was 
demographically representative of all Annenberg schools, the 12 schools that made 
up our field research sample were quite typical of schools across Annenberg and the 
system as a whole. In addition, the External Partners working with these field 
research schools were generally representative of the different types of partners 
participating in the Challenge. Our field research schools also reflected the 
demographic characteristics of the system in general. Of the 12 elementary schools 
studied for this report, six enrolled primarily African-American students, three 
enrolled primarily Latino students, three enrolled a combination of both African- 
American and Latino students (at least 85 percent of the total enrollment), and two 
enrolled a more mixed group that included between 1 5 and 30 percent white 
students. 

On average, 32 percent of students in our field research schools scored at or 
above the national average in reading on the 1999 Iowa Tests of Basic Skills (ITBS), 
and 37 percent scored at this level in math. Our field research schools ranged from 
17 to 60 percent of students at or above the national norms on the ITBS in reading 
and 16 to 78 percent of students at or above national norms in math. Average 
student enrollment for the schools was 900, ranging from 600 to 1,600 students. 

Data Collection 

Data collection took place between 1996-97 and 2000-01 school years. Baseline data 
collection took place in the 1996-97 or 1997-98 school year, depending on when the 
schools were awarded their implementation grants. The second major data collection 
point was in 1998-99. The third and last major data collection point was in the 
2000-01 school year. For the description that follows, we refer to the 1996-97 school 
year as Year 1 , 1 997-98 as Year 2, 1 998-99 as Year 3, 1 999-2000 as Year 4, and 
2000-01 as Year 5. 

Field research data collection was designed to document (a) the state of 
schools’ development on the Essential Supports at specific points in time; and (b) 
both Annenberg activities and schools’ own development activities. As noted above, 
because of the two different stages of Annenberg grant making, our documentation 
of individual schools’ development activity took place in either Years 1, 3 and 5 or 
Years 2, 3 and 5. About half of our schools fall in each category. Annenberg and 
other school development activities were documented each year. 

Our data came from several sources, including interviews with teachers, 
school administrators, Local School Council (LSC) members, Annenberg External 
Partners and Challenge staff; classroom observations and observations of Annenberg 
and other school development activities; documents related to Annenberg activity 
and school development (e.g., School Improvement Plans and reports prepared for 
the Challenge). The data we drew upon for this report included interviews and/or 
observations from 348 grade three, six and eight teachers and interviews from 225 
other school staff. We interviewed an average of 22 people at each school each year. 
We also drew from school documents and school case reports written by Project 
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researchers. Schools and staff members were promised anonymity in all reports of 
findings. 

Field research was conducted during the academic year, with interview and 
observation data typically collected between October and March. Researchers wrote 
detailed case reports for each of their schools describing their state of development at 
primary data collection years. Because of the two-stage sampling, case reports were 
written for about half of the schools for Years 1, 3 and 5 and for the other half for 
Years 2, 3 and 5. Vignettes describing each school’s development activity were also 
prepared. 

Dozens of researchers from more than eight Chicago-area colleges and 
universities assisted with the field research. A team of one lead researcher and one 
research assistant was assigned to document the development of each school in the 
study. Two-thirds of the lead researchers were faculty members at local universities. 
Two-thirds of the research assistants were graduate students at local universities. The 
research assistants had the most continuous contact with the schools (up to ten hours 
per week during periods of data collection) and the lead researchers had the primary 
responsibility for writing the case reports. The authors of this report were involved in 
each phase of the field research and also conducted interviews, observed classroom 
and development activity, and wrote case reports and vignettes. 

Data Analysis 

In this and other Project reports, the Model of Essential Supports for Students 
Learning framed our definition of school development and guided our data 
collection and analyses. The Model delineates key areas of school organization and 
practice that are strongly related to student achievement. 

Three of the authors of this report served as the primary field research data 
analysts. In their analyses they used the interviews, case studies, and documentary 
evidence gathered by field researchers for each school in the study. Indicators for high 
and low states of development on the Model of Essential Supports were used to code 
the data and determine the extent to which the field research schools developed over 
time (see Appendix A). In addition, they examined these data to identify emergent 
themes and patterns concerning the promotion and support of school development. 

Analyses of field research evidence were complicated by the contextual nature 
of the data. It was sometimes difficult to make clear-cut determinations of the levels 
of development on the Essential Supports. Therefore, the analysts independently 
rated the field research schools in terms of their strengths and weaknesses on each 
Essential Support and assessed how these levels changed over time. In general, a 
school was considered strong on an Essential Support if the evidence was indicative 
of our definition of that Support’s high state of development (see Appendix A). That 
is, there was evidence that the Support was present and reasonably well established at 
the school. A school was considered weak on a particular Support if evidence was 
indicative of our definition of a low state of development. Schools were considered 

ERJC 



179 



164 



The Chicago annenberg Challenge: Successes, Failures, and Lessons for the Future 



moderate in their development if they fell somewhere in between; that is, the 
evidence indicated that the Support’s level fell between our definitions of high and 
low states or the Support’s qualities were not reasonably established. Authors 
discussed any disagreements in their independent ratings and, where necessary, 
engaged in additional data analysis to reach consensus. 

Field researchers were asked to verily the ratings their particular schools received and 
to check the factual accuracy of information about their schools that were used in 
this report. Researchers were also asked to review the emergent themes and patterns 
of the promotion of school development and compare them to what was taking place 
in their school. 



Appendix D 

Survey Research Methods 

In the spring of 1997, 1999, and 2001, the Consortium surveyed CPS teachers, 
principals, and students in grades six through ten. Similar surveys were administered 
to teachers and students in spring 1994. In 2001, 59,663 elementary school students 
and 8,572 elementary school teachers completed surveys, representing 365 of the 
total of 492 elementary schools across the CPS system. Of the elementary school 
principals, 278 of the 492 provided usable surveys. We conducted a series of analyses 
for possible non-response bias among teachers, students, and schools in terms of basic 
school demographic characteristics (e.g., race/ethnicity, percent low income students, 
etc.). Overall, we found that the survey sample is representative of schools across 
CPS. For this report, we analyzed teacher and student survey data from 365 
elementary schools and principal survey data from 278 schools. 

Separate analyses were performed on each measure of each Essential Support 
and non-academic student outcome to determine whether there were changes in the 
measures from 1997 to 1999, from 1999 to 2001, and from 1994 to 2001 (see 
Appendix E for detailed descriptions of the measures used in these analyses). 
Annenberg elementary schools were compared to demographically similar non- 
Annenberg schools on each measure for each survey year (1994, 1997, 1999, 2001). 
Analyses were also performed to determine whether changes in the measures among 
Annenberg schools were different from changes in the measures among non- 
Annenberg schools between 1997 to 1999, 1999 to 2001, and 1994 to 2001. 

We used three-level hierarchical linear models (HLMs) to make these 
comparisons, with each survey measure acting as the dependent variable in each 
separate model. Data were structured with a case for each respondent for each survey 
year (1994, 1997, 1999 and 2001) at Levels 1 and 2, and for each school at Level 3. 
The Level 1 model was used to weight each respondent’s score, given the standard 
error in that person’s measure. Level 2 models estimated variation in the measure 
among respondents within the schools, while Level 3 models estimated differences 
across schools. The models were constructed as follows. 

Level- 1 Model 

Y = P ] *(WGT94) + P 2 *(WGT97) + P 3 *(WGT99) + P 4 *(WGT01) + e 

Level-2 Models 

P] = Bio + rl 

P 2 = B 20 + r2 
P3 ~ B 30 + r3 
P 4 = B40 + r 4 
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Level-3 Models 
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At Level 1 , a measurement model was run for each person in each school to 
determine the most accurate estimation of that person’s score on the measure, given 
the standard error of their measure (determined through Rasch analysis by their 
response pattern to the items in the question) and the average score for the school 
The dependent variable (Y) was the person’s score on the measure divided by the 
standard error on the measure. This was predicted with the inverse of the standard 
error on the measure, multiplied by dummy variables (scored one or zero) 
representing each of the survey years (WGT94, WGT97, WGT99, WGT 01 ). That 
is, if the survey response for a particular case was from the 1997 survey, the values of 
WGT94, WGT99, and WGT 01 for that case would be zero, while the value of 
WGT97 would be the inverse of the standard error of the measure for that person. 
The coefficient associated with the weight for the corresponding survey year (P p P 2 , 
P 3 P 4 ) represents the best estimate of that person’s true score on the measure in that 
year. 



At Level 2 , models were run within each school to determine the average 
score for the school on the measure for each year. Each of the coefficients from level 
one (P p P 2 , P 3 j P 4 — the best estimates of each person’s true score on the measure) is 
modeled without any predictors. The intercepts (B 10 , B 20 , B 30 B 40 ) represent the 
average score on the measure for each school for each year. 

Level 3 compared schools’ average scores (B 10 , B 20 , B 30 B 40 ) controlling for a 
number of demographic variables and a variable representing Annenberg affiliation. 
Demographic variables used for controls included the following: an index of the 
level of crime around the school neighborhood (developed from police department 
records on total incidence of crimes by location), the school’s average ITBS scores in 
1994, average social status of adults in the school neighborhood (developed from 
1990 census items on the percentage of employed persons who are managers, 
executives, etc., and the education levels of adults over 23 years old), average housing 
tenancy in the school neighborhood (from 1990 census data), average poverty in the 
school neighborhood (developed from 1990 census items on the percentage of adult 
males unemployed and the percentage of families below the poverty line), percentage 
of limited-English proficiency students in the school in 1997, percentage of low- 
income students in the school in 1997, mobility rate of students in the school in 
1997, and dummy variables representing the racial composition of the school 
(predominantly African-American, predominantly Latino, racially mixed but not 
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integrated, and mixed minority, with integrated as the excluded group), and a 
dummy variable representing small school enrollment. All of the predictor variables 
were centered on the grand mean so that the intercepts (G 100 , G 200 , G 300 G 400 ) 
represented the average score for the measure across all schools for 1994, 1997, 1999, 
and 2001, respectively. Dummy variables representing Annenberg affiliation and 
Breakthrough schools were also entered as predictors to discern any difference in the 
average score among Annenberg schools compared to other schools, controlling for 
demographic variables. These dummy variables were centered around zero according 
to their representation in the system (e.g., Annenberg = 0.3, non-Annenberg = -0.7) 
so that the equation intercepts represented the average for the system as a whole. 

The significance levels of the coefficients for the Annenberg dummy variable 
were used to determine whether Annenberg schools differed from non-Annenberg 
schools on the survey measure for each year. Contrast tests were performed to 
determine the answers to the other questions. To determine whether there was a 
significant level of change in the measure in the overall system from 1999 to 2001, a 
contrast was performed between the intercepts for 1999 and 2001 (G 300 and G 400 ). 

To determine whether Annenberg schools experienced a different rate of change in 
the measure than non-Annenberg schools, another contrast was performed between 
the coefficients associated with the Annenberg variable for 1999 and 2001 (G 3015 and 
G 40l5 ). Comparable analyses were performed on each measure to identify changes in 
measures among Breakthrough Schools and to compare changes among 
Breakthrough Schools to changes among other Annenberg schools. A 0.01 level of 
probability was used to define statistical significance, except where noted in our 
analyses of Breakthrough Schools. 

Detailed findings of these analyses are reported below in Appendix H. These findings 
are presented in terms of between group mean comparisons over time and 
standardized change units. The calculations of these change units, which are similar 
to effect size units, are described in Appendix H. 
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Appendix E 

Measures Used in Survey Analyses 

The statistical analyses performed for this report used Rasch measures of student 
social and psychological outcomes and of different elements of the Model of Essential 
Supports for Student Learning. These measures were developed by the Consortium 
on Chicago School Research from its 1994, 1997, 1999, and 2001 surveys. The 
measures consist of three to 1 5 survey items and range on a scale from 1 to 10. 
Negatively worded items or items that reflect the opposite of the phenomenon being 
measured were reversed for measure construction. This appendix provides 
definitions, internal reliability coefficients, and items for each of these measures. The 
reliability coefficients are for 1999 measures. It also provides definitions and cut 
points for the substantive scale categories of each measure. There are slight 
differences in the items used to construct these measures from year to year but these 
differences are not consequential conceptually or statistically. Additional information 
about these measures and their construction is available from the Consortium on 
Chicago School Research. 

Measures of Student Social and Psychological Outcomes 

Student Academic Engagement. This is a measure of students’ interest and 
engagement in learning, their interest in the topics they study, and their participation 
in the classroom generally. High levels indicate that students are highly engaged in 
learning. (Reliability coefficient = 0.66) 

Items: Students agree or disagree that: 

• I often count the minutes until class ends. 

• I get so interested in my work I don’t want to stop. 

• I usually look forward to class. 

• I am usually bored with what we study in this class. 

• The topics we are studying are interesting and challenging. 

• I work hard to do my best in this class. 
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Categories and Cut Points 


In this school: 


None 

0.00 to 2.41 


Students disagree or strongly disagree that they try 
hard to do their best and find their math topics 
interesting; the strongly disagree that they are not 
often bored in class, they are so interested in the work 
they don’t want to stop, and they do not often count 
the minutes until class ends. 


Limited 

2.41 to 4.65 


Students agree that they try hard to do their best; some 
agree and others disagree that their topics are 
interesting; however, they disagree that they are not 
often bored in class, they are so interested in the work 
that they don’t want to stop, and they do not often 
count the minutes until class ends. 


Moderate 

4.65 to 7.12 


Students agree or strongly agree that they work hard 
to do their best; they agree with the other items. 


High 

7.12 to 10.00 


Students strongly agree with all items on the scale. 



Student Classroom Behavior. This is a measure of students’ assessments of their 
peers’ classroom behavior with regard to how they treat each other, how often they 
disrupt class, if they have respect for each other, and if they help each other learn. 
High levels indicate that positive behaviors are more prevalent and problem 
behaviors are less so. (Reliability coefficient = 0.61). 

Items: Students agree or disagree that other students in their class: 

• Like to put others down. 

• Just look out for themselves. 

• Treat each other with respect 

• Really care about each other. 

• Get along together very well. 
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Categories and Cut Points 


In this school: 


Very negative 


Students strongly disagree with all items on the scale. 


0.00 to 2.81 




Negative 
2.81 to 5.31 


Students disagree with all items on the scale, except 
that some students strongly disagree that students do 
not disrupt class. 


Moderately positive 
5.31 to 7.81 


Attendants agree or strongly agree that students who 
do well are not made fun of, and students work 
together to solve problems, help each other learn, get 
along well, care about each other, and treat each other 
with respect; they agree that students do not look out 
just for themselves, and do not like to put others down; 
some agree and some disagree that students do not 
disrupt class. 


Very Positive 


Students strongly agree with all items on the scale. 


7.81 to 10.00 





Student Social Competence. This is a measure of students' impressions of their 
ability to help people end arguments; listen carefully; and share, help, and work well 
with each other. High levels indicate that students feel comfortably in a wide range of 
social situations. (Reliability coefficient = 0.69) 



Items: Students agree or disagree that: 

• I can always find a way to help people end arguments. 

• It's easy for me to make suggestions without being bossy. 

• I listen carefully to what other people say to me. 

■ • I'm very good at working with other students. 

• I'm good at taking turns and sharing things with others. 

• I'm good at helping people. 
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Categories 


In this school: 


None 


Students strongly disagree with all items on the scale. 


Weak 


Students disagree that they are good at helping people, 
taking turns, working with other students, they know 
how to disagree without starting a fight, listen carefully 
to what others say, and find it easy to make suggestions 
without being bossy; they disagree or strongly disagree 
that they can always find a way to help people end 
arguments. 


Moderate 


Students agree that they are good at helping people, 
taking turns, working with other students, that they 
know how to disagree without starting a fight, listen 
carefully to what others say, and find it easy to make 
suggestions without being bossy; some agree and others 
disagree that they can always find a way to help people 
end arguments. 


Strong 


Students strongly agree that they are good at helping 
people, taking turns, working with other students, they 
know how to disagree without starting a fight, listen 
carefully to what others say, and find it easy to make 
suggestions without being bossy; they agree or strongly 
agree they can always find a way to help people end 
arguments. 



Student Self-Efficacy. This is a measure of students’ level of confidence in their 
academic ability. Items ask students if they believe they can master new skills and 
succeed at even the hardest tasks if they try. High levels indicate that students feel 
they can meet high standards. (Reliability coefficient = 0.58) 

Items: Students agree or disagree that: 

• If I try hard, I can understand most of my class work. 

• I am certain I can master the skills taught in this class. 

• I can do even the hardest work in this class if I try. 

• I can do better work than I’m doing now. 

• With enough time, I can do a good job on all my work. 

• I care if I get a bad grade I this class. 
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Categories 


In this school: 


Low 


Students disagree or strongly disagree that they care if 
they get bad grades, can do better than they are now, 
and can do a good job if they have enough time; they 
strongly disagree that they can do the hardest work if 
they try, can master certain skills, and understand all 
class work if they try hard. 


Minimal 


Some students agree and some disagree that they care 
if they get bad grades and can do better than they are 
now; they disagree that they can do a good job if they 
have enough time, can do the hardest work if they 
tried, and can do better than they are; they disagree or 
strongly disagree that they can master the skills taught 
in class and understand all class work if they try hard. 


High 


Students agree or strongly agree that they care if they 
get bad grades in class, can do better than they are now, 
and can do a good job if they have enough time; they 
agree that they can do the hardest work if they try and 
are certain they can master the skills taught in class; 
some agree and others disagree that they can 
understand all class work if they try hard. 


Very High 


Students strongly agree that they care if they get bad 
grades in class, can do better than they are now, can do 
a good job if they have enough time, can do the hardest 
work if they try, and are certain they can master the 
skills taught in class; they agree or strongly agree that 
they can understand all class work if they try hard. 



Instruction Measures 

Demand for Authentic Intellectual Work. This measure assesses the extent to which 
teachers report making assignments or creating tasks that require that students 
engage in authentic intellectual work, study a topic in depth, and produce original 
work. A high score indicates a teacher who assigned lessons that require challenging 
intellectual work from students. (Reliability coefficient = 0.76). 

Items : Teachers report how often, the percentage of their lessons, or the percentage 
of classroom time the following characterize their instruction: 

• Lessons focus on studying a topic in depth, rather than covering basic facts, 
concepts, or procedures. 

• Lessons have students explaining to you or to their classmates how the topic 
relates to their personal experiences or to a problem in the contemporary 
world. 
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• Lessons require students to organize, interpret, evaluate, and use information 
to produce a piece of original work. 

• Analyzing and interpreting literature. 

• Differentiating fact from opinion. 

• Drawing inferences from expository texts. 

• Synthesizing ideas from several texts. 

• Understanding the author’s perspective. 

• Writing tasks in which students must elaborate on their ideas and 
conclusions with supporting details and evidence and organize these ideas 
into a coherent progression of sentences and paragraphs. 

• Writing tasks in which students must go beyond facts to organize and 
synthesize information, including consideration of alternative ideas. 



Category 


In this school: 


No Demand 
0.00 to 2.15 


Teachers never ask students to elaborate their ideas, or 
organize and synthesize information; spend less than 5 
percent of their class time on synthesizing ideas from 
reading, differentiating fact from opinion, drawing 
inferences, and analyzing or interpreting literature; less 
than 10 percent of their lessons deal with studying a topic 
in depth or producing original work. 


Low Demand 
2.15 to 5.15 


Teachers ask students to elaborate their ideas, and organize 
and synthesize information less than once a week; spend 
between 5 percent and 35 percent of their class time on 
synthesizing ideas from reading, differentiating fact from 
opinion, and drawing inferences; and more than 50 
percent of their time on analyzing or interpreting 
literature; between 10 percent and 50 percent of lessons 
deal with studying a topic in depth and producing original 
work. 


High Demand 

5.15 to 7.42 


Teachers ask students to elaborate their ideas, and organize 
and synthesize information once or twice a week; spend 
between 35 percent and 50 percent of their class time on 
synthesizing ideas from reading, differentiating fact from 
opinion, and drawing inferences; and more than 50 
percent of their time on analyzing and interpreting 
literature; between 50 percent and 75 percent of lessons 
deal with studying a topic in depth and producing original 
work. 


Very High Demand 
7.42 to 10.00 


Teachers ask students to elaborate their ideas, and organize 
and synthesize information nearly every day; spend more 
than 50 percent of their class time on synthesizing ideas 
from reading, differentiating fact from opinion, drawing 
inferences, and analyzing and interpreting literature; 
between 75 percent and 100 percent of lessons deal with 
studying a topic in depth and producing original work. 
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Emphasis on Writing. This measure represents the amount of writing that teachers 
ask students to do and indicates the overall emphasis that teachers place on writing in 
their teaching. (Reliability coefficient = 0.85) 

Items: Teachers report whether at least once a week they have students: 

• Write four pages or more. 

• Write one to three pages. 

• Write one page. 

• Edit/revise/publish essays. 

• Brainstorm ideas for written work. 

• Write one or two paragraphs. 



Category 


In this school: 


None 

0.00 to 1.37 


Teachers have students write one to two paragraphs once or 
twice a semester and never assign any longer writing. 


Minimal 


Teachers have students write one to two paragraphs once or 
twice a week and have students revise and edit their writing 


1.37 to 3.91 


once or twice a semester, but do no other writing. 


Moderate 


Teachers have students write one page once or twice a 
semester and one to two paragraphs once or twice a week; 


3.91 to 5.17 


they never have students write anything longer, but have the 
students edit and revise their writing once or twice a month. 


Fairly intensive 


Teachers have students write one to two paragraphs nearly 
every day, one page once or twice a month, and one to 


5.17 to 6.50 


three pages once or twice a semester; they have students edit 
and resive their written work once or twice a week. 


Intensive 


Teachers have students write four pages or more once or 
twice a semester, one to three pages once or twice a month, 


6.50 to 7.67 


and one page once or twice a week. 


Very intensive 


Teachers have their students write four pages or more once 
or twice a month, and one to three pages once or twice a 


7.67 to 10.00 


week; shorter writing is assigned almost every day. 



Didactic Instruction. This is a measure of the amount of time that teachers devote to 
whole class, teacher-centered instructional strategies including lecture, recitation, 
structured call and response, workbook exercises and other forms of individual 
student work, drill and practice, silent reading and reading aloud to other students, 
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and preparation for standardized tests. High levels indicate that teachers make greater 
use of these strategies. (Reliability coefficient = 0.75) 

Items: Teachers report how frequently they use or how important they consider 
using the following strategies in their classrooms: 

• Lecture to class for more than half the period. 

• Have students memorize facts or procedures. 

• Use highly structured call and response activities. 

• Have students complete workbook or textbook exercises in class. 

• Have students take turns reading out loud. 

• Have students read silently. 

• Consider multiple choice, true-false tests important for judging how well 
students are learning. 

• Consider short-answer tests important for judging how well students are 
learning. 



Category 


In this school: 


None 

0.00 to 1.60 


Teachers never use highly structured call and response 
exercises, lecture to the class for at least half the period, 
have students memorize facts and concepts; students read 
out loud once or twice a semester. 


Infrequent 

1.60 to 4.38 


Teachers do not use highly structured call and response 
exercises or have students memorize facts more than once 
or twice of week; they do not lecture to the class for more 
than half the period more than once or twice a month, but 
they may have students read out loud as often as once or 
twice a week. 


Regular 

4.38 to 6.95 


Teachers use call and response exercises and have students 
memorize facts and concepts once or twice a week; they 
may lecture to the class for more than half the class at least 
once or twice a month; and may have students read out 
loud nearly every day. 


Very frequent 


Teachers perform all these practices nearly every day. 


6.95 to 10.00 





Interactive Instruction. This is a measure of the amount of time that teachers devote 
to instructional strategies that involve more student-centered, interactive activities. 
These activities include having students discuss ideas in class, brainstorm, and use 
cooperative groups. High levels indicate that teachers make greater use of these strategies . 
(Reliability coefficient = 0.84) 
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Items: Teachers report how frequently they use or how important they consider 
using the following strategies in their classrooms: 

• Assign projects of at least one week’s duration. 

• Have students work in cooperative groups. 

• Have students brainstorm ideas for written work. 

• Have students discuss and debate ideas for more than half a period. 

• Engage in extended discussion around a key theme. 

• Have students talk with one another in pairs or small groups about 
something they have read. 

• Consider group projects important for judging how well students are 
learning. 

• Consider individual projects important for judging how well students are 
learning. 

• Consider student presentation of work important for judging how well 
students are learning. 

• Consider student participation in class important for judging how well 
students are learning. 

• Consider essay tests important for judging how well students are learning. 

• Consider open-ended problems important for judging how well students are 
learning. 



Category 


In this school: 


None 

0.00 to 0.71 


Teachers never have students discuss what they have read in small 
groups, use group and individual projects for judging student 
learning, or assign projects of at least one week’s duration; they have 
students work in cooperative groups once or twice a semester and 
consider student participation in class not important or somewhat 
important in judging student learning. 


Occasional 
0.71 to 3.48 


Teachers have students discuss what they have read in small groups 
not more than once or twice a month; they consider open-ended 
problems not important or somewhat important for judging 
student learning; they have students work in cooperative groups 
once or twice a week or once or twice a semester, and consider 
participation in class important in judging student learning, and 
may assign projects of one week’s duration once or twice a semester 
if at all. 


Regularly 

3.48 to 5.50 


Teachers assign project’s of one week’s duration once or twice a 
month, have students discuss what they have read in small groups, 
and use cooperative groups at least once or twice a week; they 
consider student participation in class to be very important in 
judging student learning. 


Frequent 
5.50 to 10.00 


Teachers have students engage in extended discussion around a key 
theme, and assign projects of one week’s duration at least once or 
twice a week; they engage in the other practices nearly every day, 
and consider the indicators of student learning very important. 
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Measures of Student Learning Climate 

Classroom Personalism. This measure assesses the degree to which students perceive 
that their teachers give individual attention to and are concerned about their 
students. Questions ask students if their teachers know and care about them, notice if 
they are having trouble in class, and are willing to help with academic and personal 
problems. High levels indicate that students perceive a great deal of personalized support 
from their teachers. (Reliability coefficient = 0.72) 

Items : Students agree or disagree that their teacher: 

• Relates subject matter to their personal interests. 

• Really listen to what they have to say. 

• Help them catch up if they are behind. 

• Notice if they have trouble learning something. 

• Is willing to give extra help on work if needed. 

• Believe they can do well in school. 



Category 


In this school: 


None 

0.00 to 0.80 


Students disagree or strongly disagree that their teachers believe 
they can do well in school, are willing to give extra help, notice if 
they are having trouble learning something, help them catch up if 
they are behind, and really listen to what they have to say; they 
strongly disagree that their teachers relate subject matter to their 
personal interests. 


Minimal 

0.80 to 2.93 


Some agree and others disagree that their teachers believe they 
can do well in school; all disagree that their teachers are willing 
to give extra help, notice if they are having trouble learning 
something, help them catch up if they are behind, and really 
listen to what they have to say; they disagree or strongly disagree 
that their teachers relate subject matter to their personal interests. 


Considerable 

2.93 to 5.73 


Students agree or strongly agree that their teachers believe they 
can do well in school; they agree that their teachers are willing to 
give extra help, notice if they are having trouble learning 
something, help them catch up if they are behind, and really 
listen to what they have to say; however, some agree and others 
disagree that their teachers relate subject matter to their personal 
interests. 


Strong 
5.73 to 10.00 


Students strongly agree that their teachers believe they can do 
well in school, are willing to give extra help, notice if they are 
having trouble learning something, and help them catch up if 
they are behind; they agree or strongly agree that their teacher 
listens to what they say and relate the subject matter to their 
personal interests. 
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School Safety. This measure reflects students’ sense of personal safety inside and 
outside of the school, and traveling to and from school. High levels indicate that 
students feel very safe in all these areas. (Reliability coefficient = 0.62) 

Items: Students indicate the extent to which they feel mostly safe: 

• Outside around the school. 

• Traveling between home and school. 

• In the hallways and bathrooms of the school. 

• In their classes. 



Category 


In this school: 


Not safe 
0.00 to 2.56 


Students feel somewhat or not safe in their classes and in the 
hallways and bathrooms; they do not feel safe traveling 
between home and school and outside around the school. 


Somewhat safe 
2.56 to 5.81 


Students feel somewhat or mostly safe in their classes, in the 
hallways and bathrooms, and traveling between home and 
school; they feel somewhat safe outside around the school. 


Mostly safe 
5.81 to 8.31 


Students feel very safe in their classes, and mostly or very safe 
in the hallways and bathrooms, traveling between home and 
school, and outside around the school. 


Very safe 


Students feel very safe in all these areas. 


8.31 to 10.00 





Press Toward Academic Achievement. This measure consists of students’ reports 
about the degree to which their teachers challenge them to meet high expectations 
for academic performance. Questions ask students if their teachers press them to do 
well in school, expect them to complete their homework and work hard, give praise, 
and are willing to give extra help. High levels indicate that teachers press all students 
toward academic achievement. (Reliability coefficient = 0.66) 

Items: Students agree or disagree with statements that their teachers: 

• Encourage extra work when they don’t understand something. 

• Praise their efforts when they work hard. 

• Care if they don’t do their work in the class. 

• Care if they get bad grades in the class. 

• Expect them to do their best all the time. 
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• Expect them to complete their homework every night. 

• Think it is very important that they do well in the class. 



Category 


In this school: 


None 

0.00 to 2.47 


Students disagree or strongly disagree that their teachers think 
it is important they do well, expect them to complete their 
homework and to their best, and care if they get bad grades or 
don’t do their work; they strongly disagree that their teachers 
praise them when they work hard or encourage them to do extra 
work when they don’t understand something. 


Limited 

2.47 to 4.33 


Students agree and others disagree that their teachers think it is 
important they do well, expect them to complete their 
homework and do their best, and care if they get bad grades or 
don’t do their work; they disagree that their teachers praise them 
when they work hard or encourage them to do extra work when 
they don’t understand something. 


Moderate 

4.33 to 7.40 


Students agree or strongly agree that their teachers think it is 
important they do well, expect them to complete their 
homework and do their best, do not put them down, care if they 
get bad grades or don’t do their work, and praise them when 
they work hard; they agree that their teacher encourages them to 
do extra work when they don’t understand something. 


High 

7.40 to 10.00 


Students strongly agree that their teachers think it is important 
they do well, expect them to complete their homework and do 
their best, care if they get bad grades or don’t do their work, 
praise them when they work hard, and encourage them to do 
extra work when they don’t understand something. 



Peer Support for Academic Work. This measure assesses norms among students 
with respect to their peers’ support of academic work. Questions ask students how 
many of their peers try hard to get good grades, do homework regularly, pay 
attention in class, and follow school rules. High levels indicate that students support 
each other academically. (Reliability coefficient = 0.82) 

Items: Students report the proportions of students in their class who: 

• Think doing homework is important. 

• Feel it is important to pay attention in class. 

• Feel it is important to attend all their classes. 

• Try hard to get good grades. 

• Think getting good grades is cool. 
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Category 


In this school: 


Minimal 


Few or none of the students in their class think getting good 
grades is cool, try to get good grades, attend all their classes, 


0.00 to 3.69 


pay attention in class, and think doing homework is 
important. 


Limited 


Between about half and most of the students in their class 
think getting good grades is cool; most try hard to get good 


3.69 to 5.48 


grades and attend all their classes; a few or most think doing 
homework is important and pay attention in class. 


Moderate 


Most of the students in their class try hard to get good 
grades and attend all their classes, and about half or most 


5.48 to 7.86 


pay attention in class and think doing homework is 
important. 


Strong 


All of the students in their class think getting good grades is 
cool, try hard to get good grades, and attend all of their 


7.86 to 10.00 


classes; most or all of the students in their class pay attention 
in class and think doing homework is important. 




School Leadership Measures 



Inclusive Leadership. This measure reflects teachers’ views of their principal as a 
facilitative and inclusive leader who engages parents and the community in the 
school, creates a sense of community, and is committed to shared decision making. 
High levels indicate that teachers view their principal as a leader who strongly encourages 
broad participation in school affairs. (Reliability coefficient = 0.75) 

Items: Teachers indicate their agreement or disagreement that their principal: 



Is strongly committed to shared decision making. 

Works to create a sense of community in the school. 
Promotes parent and community involvement in the school. 
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Category 


In this school: 


Negative 
0.00 to 1.72 


Teachers disagree or strongly disagree that the principal 
promotes parent and community involvement and strongly 
disagree that the principal works to create a sense of 
community in the school and is committed to shared 
decision making. 


Mixed 
1.72 to 3.94 


Teachers agree that the principal promotes parent and 
community involvement; but they disagree that the principal 
works to create a sense of community in the school or is 
committed to shared decision making. 


Positive 
3.94 to 8.96 


Teachers agree or strongly agree that the principal promotes 
parent and community involvement; they agree that the 
principal works to create a sense of community in the school 
and is committed to shared decision making. 


Very positive 


Teachers strongly agree with all items on this scale. 


8.96 to 10.00 





Joint Problem Solving. This measure indicates the extent to which teachers engage 
in public dialogue to solve problems, specifically whether they use faculty meetings to 
discuss their alternative viewpoints, and whether there are established processes for 
making public decisions. High levels indicate that there is good communication among 
faculty and that teachers work together to solve problems . (Reliability coefficient = 0.82) 

Items : Teachers agree or disagree that: 

• Faculty meetings are often used for problem solving. 

• The faculty has a good process for making group decisions. 

• Many teachers express their personal views at faculty meetings. 

• We do a good job talking through views/opinions/values. 

• When a conflict arises, we [don’t] “sweep it under the rug.” 



Category 


In this school: 


Very weak 
0.00 to 1.24 


Teachers disagree or strongly disagree with all items on the 
scale. 


Weak 
1.24 to 4.57 


Some teachers agree and others disagree that teachers sweep 
conflict under the rug; they agree that teachers do a good job 
talking through views/opinions; they agree or strongly agree 
that teachers in their school express personal views at 
meetings, have a good process for solving problems, and use 
faculty meetings for problem solving. 
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Category 


In this school: 


Strong 

4.57 to 8.57 


Teachers agree with all items on the scale. 


Very strong 
8.57 to 10.00 


Teachers strongly agree that teachers do not sweep conflict 
under the rug and do a good job talking through views and 
opinions; they agree or strongly agree that teachers in their 
school express personal views at meetings, have a good 
process for solving problems and use faculty meetings for 
problem solving. 



Teacher Influence on School Decisions. This measure indicates the extent of 
teachers’ involvement in school decision making. It assesses teachers’ influence on the 
selection of instructional materials, setting of school policy, in-service program 
planning, discretionary funds spending, and hiring of professional staff High levels 
indicate that teachers have influence on a broad range of issues at the school. (Reliability 
coefficient = 0.85) 

Items : Teachers agree or disagree that they: 

• Have some influence in hiring new professional personnel. 

• Have some influence in hiring a new principal. 

• Have some influence in planning how discretionary school funds are used. 

• Have some influence in determining the content of in-service programs. 

• Are involved in making the important decisions in this school. 

• Have some influence in setting standards for student behavior. 

• H ave informal opportunities to influence what happens here. 

• Have some influence in establishing curriculum and instmction. 

• Have some influence in determining books/instructional materials used. 



Category 


In this school: 


Minimal 

0.00 to 2.53 


Teachers have none or a little influence in determining 
instructional materials for their class and establishing 
curriculum programs; teachers disagree or strongly disagree 
that they feel comfortable voicing their concerns or are 
involved with making important decisions at the school; and 
teachers have no influence in determining in-services or 
teaching assignments, using discretionary funds, determining 
the school schedule, or hiring a new principal or personnel 
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Category 


In this school: 


Limited 

2.53 to 4.76 


Teachers have a litde or some influence in determining 
instructional materials for their class; they disagree that they 
feel comfortable voicing their concerns or are involved in 
making important decisions at the school; they have a little 
influence over establishing curriculum programs and 
determining in-services; they have none or a little influence 
over teaching assignments, using discretionary funds, and 
hiring a new principal and personnel. 


Moderate 
4.76 to 7.47 


Teachers have some or a great deal of influence in 
determining instructional materials for their class; they agree 
that they are comfortable voicing their concerns and are 
involved in making important decisions at the school; they 
have some influence over establishing curriculum programs 
and setting standards for student behavior; and they have a 
litde or some influence over teaching assignments, using 
discretionary funds, and hiring a new principal and 
personnel. 


Extensive 
7.47 to 10.00 


Teachers have a great deal of influence in determining 
instructional material for their class and setting standards for 
student behavior; teachers strongly agree that they feel 
comfortable voicing their concerns and are involved in 
making important decisions at the school; teachers have some 
or a great deal of influence in determining in-services, using 
discretionary funds, determine the school schedule, and 
hiring a new principal and personnel. 



Principal Instructional Leadership. This is a measure of teachers’ perceptions of 
their principals as instructional leaders with respect to teaching and learning 
standards, communication of a clear vision for the school, and tracking of academic 
progress. High levels indicate that teachers view their principal as very involved in 
classroom instruction. (Reliability coefficient = 0.86) 

Items : Teachers agree or disagree that their principal: 

• Carefully tracks student academic progress. 

• Understands how children learn. 

• Presses teachers to implement what they have learned in professional 
development. 

• Communicates a clear vision for the school. 

• Sets high standards for student learning. 

• Sets high standards for teaching. 

• Makes clear to staff his/her expectations for meeting instructional goals. 
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Category 


In this school: 


Weak 

0.00 to 1.80 


Teachers disagree or strongly disagree with all items on the 
scale. 


Mixed 
1.80 to 4.26 


Some teachers agree and some disagree that their principal 
makes teaching expectations clear, sets high standards for 
both teaching and student learning, and communicates a 
clear vision for the school; they disagree that their principal 
presses them to implement what they learn in professional 
development activities, understands how students learn, and 
tracks student academic progress. 


Strong 


Teachers agree with all items on the scale. 


4.26 to 7.79 




Very strong 

7.79 to 10.00 


Teachers strongly agree that their principal makes teaching 
expectations clear, sets high standards for both teaching and 
student learning, and communicates a clear vision for the 
school; they agree or strongly agree that the principal 
presses teachers to implement what they learn in professional 
development activities, understands how students learn, and 
tracks student academic progress. 



Measures of Teacher Professional Community 

Peer Collaboration. This is a measure of teachers’ reports about the level of 
cooperation and collaboration among staff. Questions ask teachers about the quality 
of the relationships among faculty, if staff coordinates teaching and learning across 
grades, and if teachers collaborate in their design of new instructional programs. High 
levels indicate that teachers have moved beyond cordial relationships with their colleagues 
to ones in which they are actively working together. (Reliability coefficient = 0.75) 

Items: Teachers agree that in this school: 

• Teachers design instructional programs together. 

• Teachers coordinate teaching with instruction at other grades. 

• Principal/teachers/staff collaborate to make the school run effectively. 

• Most teachers are cordial. 
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Category 


In this school: 


None 

0.00 to 1.18 


Teachers disagree that other teachers are cordial and disagree 
and strongly disagree that collaborative efforts make the 
school run well, and that teachers coordinate instruction 
across grades and design instructional programs together. 


Limited 
1.18 to 3.92 


Teachers agree that other teachers are cordial; some teachers 
agree and some disagree about whether collaborative efforts 
make the school run well; and all teachers disagree that 
teachers in their school coordinate instruction across grades 
and design instructional programs together. 


Significant 
3.92 to 8.63 


Teachers agree or strongly agree that other teachers are 
cordial, and agree that collaborative efforts make their school 
run well, teachers coordinate instruction across grades, and 
teachers design instructional programs together. 


Extensive 
8.63 to 10.00 


Teachers strongly agree that other teachers are cordial, and 
agree or strongly agree that collaborative efforts make their 
school run well, teachers coordinate instruction across grades, 
and teachers design instructional programs together. 



Reflective Dialogue. This is a measure of teachers 5 assessment of how often they talk 
with one another about instruction and student learning. Questions ask teachers 
about their discussion of curriculum and instruction, the school’s goals, and the best 
ways to help students learn and manage classroom behavior. High levels indicate that 
teachers frequently discuss instruction and student learning. (Reliability coefficient = 0. 
78) 

Items: Teachers report that: 

• Conversations about school’s goals occur more than twice a month. 

• Conversations about curriculum development occur more than twice a 
month. 

• Conversations about managing class behavior occur more than twice a 
month. 

• Conversations about what helps Ss learn best occur more than twice a month. 

• Teachers regularly discuss assumptions about teaching and learning. 

• Teachers share and discuss student work with other teachers. 

• Teachers talk about instruction in the teachers’ lounge. 
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Category 


In this school: 


Almost none 


Teachers disagree or strongly disagree that they talk 
informally about instruction, share and discuss student work 


0.00 to 3.6 1 


with other teachers, and discuss assumptions about student 
learning; they have conversations about how students learn 
best, managing student behavior, developing new curriculum, 
and school goals less than once a month. 


Occasional 


Teachers agree that they talk informally about instruction 
and share and discuss student work with other teachers, some 


3.61 to 5.56 


agree and some disagree that they discuss assumptions about 
student learning; they have conversations about how students 
learn best and managing student behavior two to three times 
a month, and have conversations about developing new 
curriculum and school goals less than two to three times a 
month. 


Regular 


Teachers agree that they talk informally about instruction, 
share and discuss student work with other teachers, and 


5.56 to 9.31 


discuss assumptions about student learning; they also have 
conversations with other teachers about how students learn 
best and managing student behavior more than once or twice 
a month; and have conversations about developing new 
curriculum and school goals from once to three times a 
month. 


Frequent 


Teachers strongly agree that they talk informally about 
instruction, share and discuss student work with other 


9.31 to 10.00 


teachers, and discuss assumptions about student learning; 
they also have conversations with other teachers about how 
students learn best, managing student behavior, developing 
new curriculum, and school goals almost daily. 



Focus on Student Learning. This measures the extent to which teachers feel that the 
school’s goals and actions are focused on student learning. Questions ask teachers if 
the school has well-defined learning expectations for all students, sets high standards 
for academic performance, makes decisions based on what is best for student 
learning, and works to develop students’ social skills. High levels indicate that the 
school is working to improve every student's learning. (Reliability coefficient = 0.81) 

Items: Teachers agree that this school: 

• Really works at developing students’ social skills. 

• Focuses on what’s best for student learning when making decisions. 

• Has well defined learning expectations for all students. 

• Sets high standards for academic performance. 

• Organizes the school day to maximize instructional time. 
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Category 


In this school: 


No focus 

0.00 to 3.55 


Teachers disagree or strongly disagree with all items on the 
scale. 


Not very focused 

3.55 to 4.73 


Teachers agree that the school maximizes instruction time; 
some agree and some disagree that the school sets high 
standards for academic performance, has well-defined learning 
expectations for students, and makes decisions based on what 
is best for students; they disagree that the school works at 
developing students’ social skills. 


Focused 


Teachers agree with all items on the scale. 


4.73 to 8.09 




Very focused 
8.09 to 10.00 


Teachers strongly agree that the school day is organized to 
maximize instruction time; they agree or strongly agree that 
the school sets high standards for academic performance, has 
well-defined learning expectations for students, makes 
decisions based on what is best for students, and works at 
developing students’ social skills. 



Collective Responsibility. This is a measure of teachers’ assessment of the strength of 
their shared commitment to improve the school so that all students learn. Questions 
ask teachers how many colleagues feel responsible for students’ academic and social 
development, set high standards for professional practice, and take responsibility for 
school improvement. High levels indicate a strong sense of shared responsibility among 
faculty. (Reliability coefficient = 0.92) 

Items: Most teachers in this school: 



• Feel responsible when students fail. 

• Feel responsible to help each other do their best. 

• Help maintain discipline in the entire school. 

• Take responsibility for improving the school. 

• Feel responsible for helping students develop self control. 

• Set high standards for themselves. 

• Feel responsible that all students learn. 



O 

ERIC 



203 



188 



The Chicago annenberg Challenge: Successes, Failures, and Lessons for the Future 



Category 


In this school: 


Very limited 


None or about half of the teachers feel responsible that all 


0.00 to 3.49 


students learn; some or none set high standards for 
themselves, help students with their self-control, take 
responsibility for school improvement, help discipline all 
students, help each other, and feel responsible when students 
fail. 


Limited 


About half of the teachers feel responsible that all students 


3.49 to 4.87 


learn, set high standards for themselves, and help students 
with their self-control; some or about half take responsibility 
for school improvement, help discipline all students, and help 
each other; some feel responsible when students fail. 


Fairly strong 


Most teachers feel responsible that all students learn, set high 
standards for themselves, and help students with their self- 


4.87 to 7.24 


control; about half or most take responsibility for school 
improvement, help discipline all students, help each other, 
and feel responsible when students fail. 


Strong 


Most or nearly all teachers embrace the items on this scale. 


7.24 to 10.00 





Orientation Toward Innovation. This is a measure of teachers’ perceptions of 
whether or not they are continually learning and seeking new ideas, have a “can do” 
attitude, and are encouraged to try new ideas in their teaching. High levels indicate 
that there is a strong orientation toward improvement and a willingness to be part of an 
active learning environment. (Reliability coefficient = 0.89) 

Items : Teachers agree that in this school: 

• Most teachers are willing to take risks to make the school better. 

• Most teachers are eager to try new ideas. 

• Most teachers have a “can do” attitude. 

• All teachers are encouraged to “stretch and grow.” 

• Teachers are continually learning and seeking new ideas. 

• Most teachers are really trying to improve their teaching. 
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Category 


In this school: 


Minimal 


None or some of the teachers really try to improve their 
teaching; they disagree or strongly disagree that teachers 


0.00 to 2.96 


are continually learning, are encouraged to grow, and have 
a “can do” attitude; and none or some of their teachers try 
new ideas and take risks. 


Limited 


About half of the teachers really try to improve their 


2.96 to 3.76 


teaching; some teachers agree and others disagree that 
teachers at their school are continually learning, are 
encouraged to grow, and have a “can do” attitude; only 
some of the teachers in their school try new ideas and take 
risks. 


Moderate 


About half or most of the teachers really try to improve 


3.76 to 5.68 


their teaching; they agree that teachers are continually 
learning, are encouraged to grow, and have a “can do” 
attitude; and about half of the teachers try new ideas and 
take risks. 


Extensive 


Most or nearly all of the teachers really try to improve 


5.68 to 10.00 


their teaching; they agree or strongly agree that teachers 
are continually learning, are encouraged to grow, and have 
a “can do” attitude; and most or nearly all of the teachers 
try new ideas and take risks. 



Teacher Commitment to School. This measure assesses the extent to which teachers 
feel loyal and committed to their school. Questions ask teachers if they look forward 
to going to work, would rather work somewhere else, and if they would recommend 
the school to parents. High levels indicate teachers are deeply committed to the school . 
(Reliability coefficient = 0.79) 

Items: Teachers report that they: 

• Wouldn’t want to work in any other school. 

• Would recommend this school to parents. 

• Often look forward to each working day at this school. 

• Feel loyal to this school. 
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Category 


In this school: 


None 

0.00 to 2.92 


Teachers disagree or strongly disagree with all items on the 
scale. 


Minimal 
2.92 to 4.60 


Teachers agree that they feel loyal to their school; some 
agree and some disagree that they look forward to school 
each day; all disagree that they would recommend the 
school to other parents and would not want to work at other 
schools. 


Strong 
4.60 to 8.50 


Teachers strongly agree or agree that they feel loyal to their 
school; agree that they look forward to school each day, 
would recommend the school to other parents, and would 
not want to work at other schools. 


Very strong 
8.50 to 10.00 


Teachers strongly agree that they feel loyal to their school; 
agree or strongly agree that they look forward to school 
each day, would recommend the school to other parents, 
and would not want to work at other schools. 



Measures of Parent and Community Involvement 

Teacher Outreach to Parents. This is a measure of the school’s effort to work with 
parents to develop common goals, good communication, and strengthen student 
learning. Questions ask teachers about their efforts to understand parents’ problems, 
invite parents to visit classrooms, seek parents’ feedback, and build relations with 
parents. High levels indicate mutually supportive relationships among parents and 
teachers. (Reliability coefficient^. 85) 

Items : Teachers agree that at this school: 

• Teachers work closely with parents to meet students’ needs. 

• Parents are invited to visit classrooms. 

• Teachers communicate with parents about how they can help their kids 
learn. 

• Teachers communicate to parents support needed to advance school mission. 

• Teachers encourage feedback from parents and the community. 

• The principal pushes teachers to communicate regularly with parents. 

• Teachers really try to understand parents’ problems and concerns. 

• Parents are greeted warmly when they call or visit. 
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Category 


In this school: 


None 

0.00 to 1.54 


Teachers disagree or strongly disagree with all items on the 
scale. 


Moderate 
1.54 to 3.42 


Teachers agree that parents are greeted warmly when they visit 
the school, teachers try to understand parents’ problems, the 
principal pushes teachers to communicate with parents, and 
the school encourages feedback from parents; some agree and 
some disagree that the school works at communicating with 
parents about advancing the school mission and helping 
children learn; they disagree that parents are invited to the 
classroom or teachers work closely with parents. 


Significant 


Teachers agree with all items on the scale. 


3.42 to 6.84 




Broad 


Teachers strongly agree or agree with all items on this scale. 


6.84 to 10.00 





Parent Involvement in School. This is a measure of teachers' reports on the level of 
parent involvement, and support for the school. Questions ask teachers how often 
parents pick up report cards; attend parent-teacher conferences and school events; 
volunteer to help in the classroom; and participate in fund-raising events. High levels 
indicate that many parents are actively engaged with the school. (Reliability 
coefficient^. 82) 

Items : Teachers report that of the students they taught this year, most of the parents: 

• Volunteered to help in the classroom. 

• Helped raise funds for the school. 

• Attended school-wide special events. 

• Attended parent/ teacher conferences when requested. 

• Showed up for school events or conferences intended for them. 

• Picked up their child’s last report card. 
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Category 


In this school: 


Minimal 
0.00 to 1.97 


Teachers report none or about half of the parents picked up 
their child’s report cards and attended school events; none or 
some attended parent/ teacher conferences and special school- 
wide events; none of the parents helped raise funds for the 
school or volunteered in the classroom. 


Limited 

1.97 to 4.09 


Teachers report about half or most of the parents picked up 
their child’s report cards and attended school events; some or 
about half attended parent/ teacher conferences; some 
attended special school-wide events and helped raise funds for 
the school; none of the parents volunteered in the classroom. 


Moderate 

4.09 to 6.97 


Teachers report most or nearly all parents picked up their 
child’s report cards and attended school events and 
parent/ teacher conferences; some or about half attended 
special school-wide events and helped raise funds for the 
school; only some volunteered in the classroom. 


High 

6.97 to 10.00 


Teachers report nearly all parents picked up their child’s 
report cards and attended school events and parent/ teacher 
conferences; most or nearly all attended special school-wide 
events and about half to nearly all helped raise funds for the 
school and volunteered in the classroom. 



Teachers’ Use of Community Resources. This is a measure of the extent of teachers’ 
use of the local community as a resource in both their teaching and in their efforts to 
understand students better. Questions ask teachers how often they invite guest 
speakers from the community to the school, consult community members, and use 
examples from the community in their teaching. High levels indicate that teachers are 
taking greater advantage of community resources and making more of an effort to engage 
the communities where their students live. (Reliability coefificient=0.68) 

Items: Teachers report that at least three times this school year, they have: 

• Brought in a guest speaker from the school’s community. 

• Taken students on a field trip in the school’s community. 

• Collected materials to use in class from community businesses. 

• Consulted with community members to better understand students. 

• Told students about community agencies that can help with problems. 

• Used people or events from the community as examples. 
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Category 


In this school in the last year: 


No use 
0.00 to 2.80 


Teachers report they used people/events from the community 
as an example and told students about community agencies 
once or twice, or never; never consulted with community 
members to understand students better, collected materials 
from the business community for class, took students on a 
field trip or brought in guest speakers from the community. 


Occasional 
2.80 to 5.81 


Teachers report they used people/events from the community 
as an example and told students about community agencies 
once to four times; consulted with community members to 
better understand students and collected materials from 
community businesses for class once or twice; took students 
on a field trip or brought in guest speakers from the school 
community once or twice, or never. 


Frequent 

5.81 to 7.74 


Teachers report they used people/events from the community 
as an example and told students about community agencies 
five to nine times; consulted with community members to 
better understand students and collected materials from 
community businesses for class three to four times; took 
students on a field trip or brought in guest speakers from the 
school’s community once or twice. 


Extensive 
7.74 to 10.00 


Teachers report they used people/events from the community 
as an example and told students about community agencies 
more than 10 times; consulted with community members to 
better understand students and collected materials from 
community businesses for class more than five times; took 
students on a field trip or brought in guest speakers from the 
school’s community more than three or four times. 



Teachers’ Ties to Community. This measure assesses the extent to which teachers 
interact with the school’s surrounding community, specifically how often they visit 
students’ homes, shop, and attend religious and recreational events in the community 
where students are present. High levels indicate that teachers are more involved in the 
school's surrounding community and therefore more able to play an extended role in 
students' lives. (Reliability coefficient=0.66) 

Items : Teachers report that at least two to three times a month they: 

• Visit students’ homes. 

• Attend religious services where the students attend. 

• Attend civic and recreational events in the school’s community. 

• Shop in the school’s community. 

• They have friends who live in the school’s community (%yes). 
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Category 


In this school: 


Weak 


Some teachers have friends who live in the community; they 
shop in the school community less than once a month, but 


0.00 to 1.90 


never attend recreational activities in the school community 
or religious services where students attend, or visit the homes 
of students. 


Slight 


Teachers reported they have friends who live in the school 
community; they shop in the school community once or 


1.90 to 6.20 


twice a month; attend recreational activities in the school 
community two or three times a month; and attend religious 
services where students attend and visit the homes of students 
less than once a month. 


Strong 


Teachers reported they have friends who live in the 
community; they shop in the school community fewer than 


6.20 to 8.68 


two or three times a month; attend recreational activities in 
the school community less than once a month; but never 
attend religious services where students attend or visit the 
homes o students. 


Very strong 


Teachers reported they have friends who live in the school 


8.68 to 10.00 


community; they shop in the school community almost 
daily; attend recreational activities in the school community 
at least once or twice a week; and attend religious services 
where students attend and visit the homes of students at least 
two or three times a month. 



Teachers’ Knowledge of Students’ Culture. This is a measure of teachers’ reports 
about their efforts to better understand their students. Questions ask teachers how 
many of their colleagues talk with students about their culture and home lives, and 
whether they know about the issues facing the surrounding community. High levels 
indicate that many teachers are committed to learning more about their students and the 
community where they live. (Reliability coefficient=0.70) 

Items : Most teachers in this school: 

• Read books/watch documentaries to learn about S’s cultural backgrounds. 

• Talk with students about their lives at home. 

• Talk with students about their cultures. 

• Are knowledgeable of issues and concerns in the community. 
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Category 


In this school: 


Minimal 


None or only some of the teachers engage in these activities. 


0.00 to 2.92 




Limited 


About half of the teachers know about community issues; 
some or about half talk with students about their homes and 


2.92 to 4. 17 


cultures; and some try to learn about students’ cultural 
backgrounds. 


Significant 


Most teachers know community issues; about half or most 
talk with students about their homes and cultures; and about 


4.17 to 6.11 


half try to learn about students’ cultural backgrounds. 


Extensive 


Most or nearly all teachers engage in these activities. 


6.11 to 10.00 





Human and Social Resources in Students 1 Community. This is a measure of 
students’ assessments of the level of their trust in and reliance upon neighbors and 
community members, and whether they feel adults in the community know and care 
about them and each other. Questions ask students if adults know who the local 
children are, make sure they are safe, and can be trusted. High levels indicate that 
many students can turn to community resources for support . (Reliability coefficient = 

0.75) 

Items: Students agree or disagree that in their neighborhood: 

• Neighbors get together to deal with problems. 

• People can be trusted. 

• You can count on adults to see that children are safe. 

• The equipment and building in the park/playground are well kept. 

• There are adults that children can look up to. 

• Adults know who the local children are. 

• During the day, it is safe for children to play in the park. 

• Someone cares about what happens here. 
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Category 


In this school: 


None 


Students disagree or strongly disagree that people in the 
neighborhood care about what happens there; they strongly 


0.00 to 1.91 


disagree with the remaining items on the scale. 


Scarce 


Students agree and others disagree that people in the 


1.91 to 4.56 


neighborhood care about what happens there; they disagree 
that the parks are safe for kids to play in during the day and 
there are adults in the neighborhood who know the local 
kids and whom the kids can lookup to; they disagree or 
strongly disagree that adults make sure neighborhood kids 
are safe, people in the neighborhood can be trusted, and 
neighbors deal with any problems in the neighborhood. 


Some 


Students agree or strongly agree that people in the 


4.56 to 8.09 


neighborhood care about what happens there; they agree 
that the parks are safe for kids to play in during the day and 
there are adults in the neighborhood who know the local 
kids and whom the kids can look up to; some agree and 
others disagree that adults make sure neighborhood kids are 
safe, people in the neighborhood can be trusted, and the 
neighbors deal with any problems in the neighborhood. 


Many 


Students strongly agree that people in the neighborhood 
care about what happens there, the parks are safe for kids to 


8.09 to 10.00 


play in during the day, and there are adults in the 
neighborhood who know the local kids and whom the kids 
can look up to; they agree or strongly agree that adults 
make sure neighborhood kids are safe, people in the 
neighborhood can be trusted, and the neighbors deal with 
any problems in the neighborhood. 




Measures of Relational Trust 



Teacher-Principal Trust. This measure assesses the extent to which teachers feel their 
principal respects and supports them. Questions ask teachers if the principal looks 
out for their welfare, has confidence in their expertise, and if they respect the 
principal as an educator. High levels indicate that teachers share deep mutual trust and 
respect with the principal (Reliability coefficient^. 89) 

Items: Teachers agree that: 

• It’s OK to discuss feelings and worries with the principal. 

• The principal looks out for the personal welfare of the faculty. 

• They trust the principal at his or her word. 

• The principal is an effective manager. 
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• The principal places the needs of children before personal interests. 

• The principal has confidence in the expertise of teachers. 

• The principal takes personal interest in faculty professional development. 

• They feel respected by their principal. 



Category 


In this school: 


No trust 


Teachers feel respected by their principal not at all; they 
disagree or strongly disagree that they respect their 


0.00 to 2.58 


principal as an educator, that the principal takes an interest 
in teachers’ professional development, has confidence in 
teachers’ expertise, places students’ needs before personal 
needs, is an effective manager or looks out for teachers’ 
welfare, that they trust their principal, or it is OK to discuss 
worries with the principal. 


Minimal trust 


Teachers feel respected by their principal a little; they 
disagree that they respect their principal as an educator, that 


2.58 to 4.50 


the principal takes an interest in teachers’ professional 
development, has confidence in teachers’ expertise, places 
students’ needs before personal needs, is an effective 
manager, looks out for teachers’ welfare, that they trust their 
principal, and it is OK to discuss worries with the principal. 


Strong trust 


Teachers feel respected by the principal some or to a great 
extent; they agree that they respect their principal as an 


4.50 to 7.67 


educator, that the principal takes an interest in teachers’ 
professional development, has confidence in teachers’ 
expertise, places students’ needs before personal needs, is an 
effective manager, looks out for teachers’ welfare, that they 
trust their principal, and it is OK to discuss worries with the 
principal. 


Very strong trust 


Teachers feel respected by their principal to a great extent; 
they strongly agree that they respect their principal as an 


7.67 to 10.00 


educator, that the principal takes an interest in teachers’ 
professional development, has confidence in teachers’ 
expertise, places students’ needs before personal needs, is an 
effective manager, looks out for teachers’ welfare, that they 
trust their principal; they agree or strongly agree that it is 
OK to discuss worries with the principal.. 



Teacher-Teacher Trust. This measures the extent to which teachers feel they have 
mutual respect for each other, for those who lead school improvement efforts, and 
for those that are experts at their craft. Questions also ask teachers if they feel 
comfortable discussing their feelings and worries and really care about each other. 
High levels indicate teachers trust and respect each other. (Reliability coefficient^. 82) 
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Items: Teachers agree that in this school: 

• Most teachers really care about each other. 

• Teachers trust each other. 

• It’s OK to discuss feelings and worries with other teachers. 

• Teachers respect colleagues who lead school improvement efforts. 

• Teachers respect those colleagues who are expert at their craft. 

• They feel respect from other teachers. 



Category 


In this school: 


No trust 


Teachers feel respected by none or some of the other 
teachers; they disagree or strongly disagree that 


0.00 to 3.57 


teachers respect colleagues who are expert at their 
craft or who lead school improvement efforts, it is 
OK to discuss worries with other teachers, and 
teachers trust each other; and they feel that none of 
the teachers care about each other. 


Minimal trust 


Teachers feel respected by some of the other 
teachers; they agree that teachers respect colleagues 


3.57 to 5.56 


who are expert at their craft or who take the lead in 
school improvement efforts, and it is OK to discuss 
worries with other teachers; some agree and some 
disagree that teachers trust each other at this school; 
and none to some of the teachers care about each 
other. 


Strong trust 


Teachers feel respected by other teachers to a great 
extent; they agree that teachers respect colleagues 


5.56 to 7.06 


who are expert at their craft or who take the lead in 
school improvement efforts, it is OK to discuss 
worries with other teachers and teachers trust each 
other; and they feel that about half of the teachers in 
the school care about each other. 


Very strong trust 


Teachers feel respected by other teachers to a great 
extent; they strongly agree that teachers respect 


7.06 to 10.00 


colleagues who are expert at their craft or who take 
the lead in school improvement efforts; they agree or 
strongly agree it is OK to discuss worries with other 
teachers and that teachers trust each other; and they 
feel most or nearly ail teachers in the school care 
about each other. 



Student-Teacher Trust. This is a measure of students’ perceptions about the quality 
of their relationships with teachers. Questions ask students if teachers care about 
them, keep promises, listen to their ideas, and try to be fair. High levels indicate that 
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there is trust and open communication between students and teachers . (Reliability 
coefficient^. 78) 

Items : Students agree that their teachers: 

• Always keep their promises. 

• [Do not] punish kids without knowing what happened. 

• Make them feel safe and comfortable. 

• Always try to be fair. 

• Will always listen to students' ideas. 

• [Do not] get mad whenever the students make a mistake. 

• Really care about students. 

• [Do] care about what the students think. 

• Have a good reason when they tell the students not to do something. 



Category 


In this school: 


No trust 
0.00 to 1.34 


Students disagree that their teacher has a good reason for 
telling them not to do something, cares about them and 
what they think, does not get mad when they make 
mistakes, will always listen to students' ideas, always tries to 
be fair, makes them feel safe and comfortable, and can be 
trusted; they disagree or strongly disagree that their teacher 
does not punish students without knowing what happened 
and keeps his or her promises. 


Minimal trust 
1.34 to 2.84 


Some students agree and others disagree that their teacher 
has a good reason for telling them not to do something, and 
cares about what they think; they disagree that their teacher 
really cares about them, gets mad when they make mistakes, 
will always listen to students' ideas, always tries to be fair, 
makes them feel safe and comfortable, can be trusted, does 
not punish students without knowing what happened, and 
keeps his or her promises. 


Strong trust 

2.84 to 6.42 


Students agree that their teacher has a good reason for 
telling them not to do something, and cares about what they 
think, does not get mad when they make mistakes, will 
always listen to students' ideas, always tries to be fair, makes 
them feel safe and comfortable, and can be trusted; some 
agree and others disagree that their teacher does not punish 
students without knowing what happened and keeps his or 
her promises. 
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Category 


In this school: 


Very strong trust 
6.42 to 10.00 


Students strongly agree that their teacher has a good reason 
for telling them not to do something, and cares about what 
they think, does not get mad when they make mistakes, will 
always listen to students’ ideas, always tries to be fair, makes 
them feel safe and comfortable, and can be trusted; agree or 
strongly agree that their teacher does not punish students 
without knowing what happened and keeps his or her 
promises. 



Teacher-Parent Trust. This is a measure of teachers’ perception of the degree of 
mutual respect between themselves and parents, and their support of each other’s 
efforts to improve student learning. Questions ask teachers if they consider 
themselves partners with parents in educating children, if they receive strong parental 
support, if the school staff works hard to build trust with parents, and if teachers 
have respect for parents. High levels indicate mutually supportive relationships among 
parents and teachers. (Reliability coefificient=0.58) 

Items : At this school, teachers agree or disagree that: 

• Most students’ parents do their best to help their children learn. 

• Most teachers feel good about parents’ support for their work. 

• Most students’ parents support my teaching efforts. 

• Teachers and parents think of each other as partners in educating kids. 

• It [isn’t] difficult overcoming cultural barriers between teachers and parents. 

• Parents have confidence in teachers’ expertise. 

• Staff members work hard to build trusting relationships with parents. 

• Teachers feel respect from the parents of their students. 
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Category 


In this school: 


No trust 


Teachers respect and feel respected by parents not at all or a 
little; they disagree or strongly disagree that talking with 


0.00 to 2.03 


parents helps them understand students better, there is no 
conflict between parents and teachers, and teachers and 
parents are partners in educating children; none of the 
parents support their teaching efforts or do their best to help 
their children learn, and none of the teachers care about the 
community or feel good about parental support. 


Minimal trust 


Teachers respect and feel respected by parents to some 


2.03 to 3.14 


extent; they agree that talking with parents helps them 
understand students better; but some agree and some 
disagree that there is no conflict between parents and 
teachers, and that teachers and parents are partners in 
educating children; none to some of the parents support 
their teaching efforts or do their best to help their children 
learn, and none to some of the teachers care about the 
community or feel good about parental support. 


Strong trust 


Teachers respect and feel respected by parents to a great 
extent; they agree or strongly agree that talking with parents 


5.14 to 8.11 


helps them understand their students better, and agree that 
there is no conflict between parents and teachers, and that 
teachers and parents are partners in educating children; 
about half of parents support their teaching efforts or do 
their best to help their children learn, and about half of 
teachers care about the community or feel good about 
parental support. 


Very strong trust 


Teachers respect and feel respected by parents to a great 
extent; they strongly agree that talking with parents helps 


8.11 to 10.00 


them understand students better, there is no conflict 
between parents and teachers, and that teachers and parents 
are partners in educating children; most or nearly all 
parents support their teaching efforts and help their children 
learn, and most or nearly all teachers care about the 
community or feel good about parental support. 



Measure of School Instructional Program Coherence 

This is a measure of the degree to which teachers feel the programs at their school are 
coordinated with each other and with the school’s mission. Questions ask teachers if 
instructional materials are consistent within and across grades and if there is 
sustained attention to quality program implementation. High levels indicate that the 
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school's programs are coordinated and consistent with its goals for student learning 
(Reliability coefficient^. 75) 

Items: Teachers agree that at this school: 

• You can see continuity from one program to another. 

• Many special programs [do not] come and go. 

• Once we start a new program, we follow-up with it. 

• Curriculum and instruction are well coordinated across grades. 

• We [do not] have so many programs that I can't keep track. 

• Curriculum and instruction are consistent among teachers in same grade. 

• Coordination/focus of instruction has changed for better in last two years. 



Category 


In this school: 


None 


Teachers believe the focus of the instructional programs has 
changed for the worse; they strongly disagree with all other 


0.00 to 1.50 


items on the scale. 


Little 


Teachers believe that there has been no change in the focus 
of instructional programs in their school; some agree and 


1.50 to 4.70 


some disagree that changes in the school promote the 
school's goals for student learning; they disagree with the 
remaining items on the scale. 


Moderate 


Teachers agree with the items on this scale and believe that 
the focus of instructional programs has changed for the 


4.70 to 8.20 


better. 


Strong 


Teachers strongly agree with the items on this scale and 
believe that the focus of instructional programs has changed 


8.20 to 10.00 


for the better. 



Teacher Professional Development and Support for Change Measures 

Teacher Participation in Professional Development. This is an indicator of the 
frequency with which teachers participated in formal, planned professional 
development activity. Items used for this indicator asked respondents to report the 
number of times during the school year that they participated in professional 
development activity provided by a variety of sources. These include activities and 
courses organized by teachers' own schools, networks of teachers from other schools, 
outside professional groups or organizations, colleges and universities, the Chicago 
Public Schools, and the Chicago Teachers Union. 
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The items that compose this indicator do not form a scale like other measures used in 
this report. There are no categories or cut points. High levels of this indicator mean 
frequent participation in professional development activity across different sources. 



Quality Professional Development. This is a measure of teachers’ assessment of the 
degree to which professional development has influenced their teaching, helped them 
understand students better, and provided them with opportunities to work with 
colleagues and teachers from other schools. High levels indicate that teachers are 
involved in sustained professional development focused on important school goals. 
(Reliability coefficient=0.84) 

Items: Teachers agree that at this school their professional development experiences: 

• Included opportunities to work with teachers from other schools. 

• Included opportunities to think about, try, and evaluate new ideas. 

• Addressed the needs of the students in my classroom. 

• Deepened my understanding of subject matter. 

• Helped me understand my students better. 

• Have been sustained and coherently focused rather than short-term and 
unrelated. 

• Included opportunities to work with colleagues in my school. 

• Let me to make changes in my teaching. 

• Have been closely linked to my school’s school improvement plan (SIP). 



Category 


In this school: 


Very low quality 

0.00 to 1 .95 


Teachers disagree or strongly disagree that their 
professional development experiences were closely connected 
to the SIP, led to changes in their teaching, provided 
opportunities to work with colleagues, or provided a deeper 
understanding of the subject matter; they strongly disagree 
that it shifted their approach to teaching, included enough 
time to think about and judge the new ideas, or provided 
opportunities to work with teachers from other schools. 


Low quality 

1.95 to 4.22 


Some teachers agree and others disagree that their 
professional development experiences were closely connected 
to the SIP; teachers disagree that it led to changes in their 
teaching, provided opportunities to work with colleagues, or 
helped them understand their students better; they disagree 
or strongly disagree that it shifted their approach to 
teaching, included enough time to think about and judge 
the new ideas, or provided opportunities to work with 
teachers from other schools. 
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Category 


In this school: 


High quality 

4.22 to 7.42 


Teachers agree that their professional development 
experiences were closely connected to the SIP, provided 
opportunities to work with other colleagues, were sustained 
and focused, helped them understand their subject matter 
better, addressed students’ needs, and included enough time 
to think about and judge the new ideas; some agree and 
others disagree that it provided opportunities to work with 
teachers from other schools. 


Very high quality 
7.42 to 10.00 


Teachers strongly agree that their professional development 
experiences were closely connected to the SIP, provided 
opportunities to work with other colleagues, were sustained 
and focused, and addressed students’ needs; they agree or 
strongly agree it shifted their approach to teaching, included 
enough time to think about and judge the new ideas, and 
provided the opportunity to work with teachers from other 
schools. 



Support for Change. This is a measure of the level of support for change that 
teachers receive from their principal and colleagues. Questions ask teachers if their 
principal encourages them to take risks and try new methods of instruction, and to 
assess whether the faculty as a whole embraces change initiatives. High levels indicate 
a school-wide environment supportive of change. (Reliability coefficient^. 82) 

Items: Teachers agree that in this school: 

• Changes [do not] involve only a few teachers. 

• Teachers receive adequate professional development for changes they 
introduce. 

• Changes gain support among teachers. 

• Changes receive strong support from the principal. 
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Category 


In this school: 


None 


Teachers disagree or strongly disagree with all items on the 
scale. 


0.00 to 1.81 




Minimal 


Some teachers agree and some disagree that the principal 
encourages them to try new methods and is willing to make 


1.81 to 4.29 


changes, and that changes introduced at the school receive 
strong support from the principal; they disagree that the 
principal encourages teachers to take risks, changes 
introduced at the school gain support among teachers, 
adequate professional development is provided for changes 
that are made, and changes involve many teachers. 


Moderate 


Some teachers agree or strongly agree that the principal 
encourages them to try new methods and is willing to make 


4.29 to 7.43 


changes, and that changes introduced at the school receive 
strong support from the principal; they agree that the 
principal encourages teachers to take risks, changes 
introduced at the school gain support among teachers, 
adequate professional development support is provided for 
changes that are made, and changes introduced at the school 
involve many teachers. 


Strong 


Teachers strongly agree that the principal encourages them 
to try new methods, is willing to make changes, encourages 


7.43 to 10.00 


teachers to take risks, and that changes introduced at the 
school receive strong support from the principal and gain 
support among teachers, and that adequate professional 
development support is provided for changes that are made; 
they agree that changes introduced at the school involve 
many teachers. 



Appendix F 

The Productivity Index 1 



To assess differences in student academic achievement between Annenberg and 
demographically similar non-Annenberg schools, we used the Consortium's 
productivity index. The index estimates six-year trends in ITBS reading and math 
scores (1995 to 2001) using hierarchical linear modeling and taking into account 
four basic elements: (a) initial achievement status, (b) base gain, (c) input trend, and 
(d) gain trend. The productivity index is the gain trend adjusted for the other three 
elements. Since gain trend is correlated with initial status, base gain, and especially 
with the input trend, adjusting the gain trend for these three factors takes into 
consideration schools' starting points and produces a more powerful indicator than 
the unadjusted gain trend. 2 Taking into account demographic characteristics (listed 
in Appendix G), differences between Annenberg schools and non-Annenberg schools 
and between Breakthrough schools and non-Breakthrough Annenberg schools in 
three adjusted gain trends were compared to zero to determine statistical significance. 

Calculation of the productivity index begins with identifying a stable group 
of students; that is, a specific group of students of the same age or grade level who 
received instruction for at least one full academic year in a school. The learning gain 
for each student in this group in each year is computed by subtracting the output 
status — the student's ITBS test score at the end of the academic year — from the 
input status — the student's ITBS test score from the preceding year. 

Initial status refers to the average of these students' spring 1995 test scores. 
Base gain begins with the base period of the 1995-96 school year and is calculated as 
the difference in the initial status compared to the students' test scores in the spring 
of 1996. The base gain shows how much knowledge and skill students had gained at 
the end of a year of instruction. The input trend shows the variation in a student 
groups' input status from 1995 through 2000. The output trend shows the variation 
in their output status from 1996 through 2001. The resulting gain trend varies with 
initial and output status. 

Using the productivity index allowed us to examine student performance 
across the years in ways that adjust for changes in CPS testing practices and related 
policies that affect scores. For example, a common inclusion standard for bilingual 
education students is used across the entire time period of the index even though 
CPS policy of whose scores are included in school averages has changed over that 
time period. In addition, comparative analyses of student achievement using the 



1 See Rosenkranz (2002) and Easton, Rosenkranz, and Bryk (2001) for details on recent CPS ITBS 
trends and the construction and use of productivity index. 

2 For detail on the development of the productivity index see Bryk, Thum, Easton, and Luppescu 
(1998). 
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productivity index group children by age rather than grade level. This allowed us to 
minimize effects of the CPS retention policy on our results. 

As we explained earlier, we use the productivity index to conduct this report’s 
comparative analysis of student achievement in different groups of schools. However, 
because the productivity index is difficult to interpret, we present figures describing 
the 1TBS trends in grade equivalent gains, a reporting practice also used in the 
Consortium’s annual review of test score trends. Grade equivalents align students’ 
raw scores with a standard national average score at a particular grade. For spring 
testing, as is done in CPS, this standard score is the equivalent of the grade level plus 
eight months. Therefore, a grade equivalent of 4.8 is equal to the test score national 
average for fourth graders. If fourth graders in Chicago averaged a 3.8 grade 
equivalent in 1992, they scored one year below what students achieved nationally. 

Using yearly GE gains to describe achievement trends is complicated because CPS 
used three different ITBS forms between 1993 and 2001. The different forms and 
the years in which they were administrated are as follows: Form K (1993, 1995, 
2000); Form L (1994, 1996, 1998, 2001); Form M (1997, 1999). Our measures of 
GE gains do not take into account any effect of using different forms from year to 
year. As can be seen in the findings, there are yearly fluctuations in GE gains that 
coincide with the use of different forms. One way to take into account the use of 
different forms is to compare GE gains in years that have the same form-to-form 
changes. So when considering the findings in Part Two, it may be useful to compare 
gains in 1994, 1996, and 2001, the years in which Form L was administered and the 
years for which gains are computed on the basis of changes from Form K to Form L. 
Likewise, it may be useful to compare gains in 1997 and 1999, the years in which 
Form M was administered and the years for which gains are computed on the basis of 
changes from Form L to Form M. 
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Appendix G 

Detailed Results of ITBS Analyses 

Tables Gl and G2 present trends in reading and math one-year grade equivalent 
(GE) gains made by students in Annenberg schools and those in demographically 
similar schools not in Annenberg networks. Details on the measurement of student 
achievement using ITBS scores are contained in Part One of the report and in 
Appendix F. 

Tables G3 through G6 contain the results of productivity analyses in reading and 
math achievement for the period before the Annenberg Challenge and the period of 
the Challenge. Details on how these analyses were conducted are contained in 
Appendix F. 

Variables contained in each report of parameter estimates are defined as follows. 

• Annenberg School is a dummy variable coded 0 and 1 so that the coefficient 
gives the difference between Annenberg and non-Annenberg schools. 

• Breakthrough School is a dummy variable coded 0 and 1 so that the 
coefficient gives the difference between Breakthrough and other Annenberg 
schools. 

The following variables are for the year 2000. 

• Crime rate is the composite crime rate in the neighborhood of the school. 

• Home tenancy is the average number of years of tenancy per home owner in 
the school’s neighborhood. 

• Poverty concentration is the mean concentration of poverty in the school’s 
neighborhood. 

• Social status is the mean social status in the school’s neighborhood, not 
considering income. 

• Percent LEP students is the percentage of students in the school with limited 
English proficiency. 

• Percent low-income students is the percentage of low-income students in the 
school, based on eligibility for free- or reduced-priced lunch. 

• Predominantly African-American is a dummy variable (0 and 1) indicating 
that the school’s enrollment was at least 85 percent African-American. 

• Predominantly Latino is a dummy variable (0 and 1) indicating that the 
school’s enrollment was at least 85 percent Latino. 

• Predominantly minority is a dummy variable (0 and 1) indicating that the 
school’s enrollment was at least 85 percent African-American and Latino 
combined. 
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• Mixed race is a dummy variable (0 and 1) indicating that the school’s 
enrollment is between 15 percent and 30 percent white. 

• Small school is a dummy variable (0 and 1) indicating that the school’s 
enrollment is not more than 350 students. 

Student mobility is the number of students transferring in plus number of students 
transferring out of school, divided by beginning enrollment. 
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Table G1 Trends in Reading and Math GE Gains by Year and Grade for Students in Annenberg Schools 

Trends in Reading Gains, GE 
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Table G4. Productivity Analyses: Average Adjusted Gain Trend for Reading Achievement on the ITBS, 1996-97 to 2000-01 
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Table G6. Productivity Analyses: Average Adjusted Gain Trend for Math Achievement on the ITBS, 1996-97 to 2000-01 
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Appendix H 

Detailed Results of Survey Analyses 

The tables below present findings from statistical analyses that address the following 
questions: (1) Did CPS elementary schools change on measures of the Essential 
Supports between 1994 and 2001? (2) Were Annenberg elementary schools any 
different on measures of the Essential Supports than demographically similar non- 
Annenberg schools? (3) Were Annenberg Breakthrough elementary schools any 
different on measures of the Essential Supports than comparable non-Breakthrough 
Annenberg schools? 

Table HI reports the means and standard deviations of all measures of the 
Essential Supports for all elementary schools in CPS in 1994, 1997, 1999, and 2001. 
Table H2 reports the means of all measures of the Essential Supports for Annenberg 
and demographically similar non-Annenberg schools. This table also identifies 
differences between Annenberg and non-Annenberg schools that are statistically 
significant at the 0.01 level. 

Table H3 reports standardized change unit differences across years on 
measures of the Essential Supports among Annenberg elementary schools. These 
differences were calculated by subtracting the base year mean from the comparison 
year mean for Annenberg schools and dividing the difference by the standard 
deviation of the system mean (all elementary schools) for the base year. For example, 
the standardized change unit difference for student academic engagement in 
Annenberg schools between 1994 and 2001 was calculated by subtracting the 2001 
mean for student engagement in Annenberg schools from the 1994 mean for student 
engagement in Annenberg schools and then dividing the difference by the standard 
deviation of the 1994 mean for student engagement for all schools in the system. 

Table H4 reports the standardized change unit differences between 
Annenberg and demographically similar non-Annenberg schools in 1994, 1997, 
1999, and 2001 in comparison to the system mean in the baseline year. The baseline 
year is either 1994 or 1997, depending on when data on the measure were first 
available. For Annenberg schools, differences were calculated by subtracting the 
system mean in the baseline year (either 1994 or 1997) from the Annenberg school 
mean in a particular year (e.g., 1994, 1997, 1999, 2001) and then dividing that 
difference by the standard deviation of the system mean for the baseline year. For 
example, the standardized change unit difference for student academic engagement 
between Annenberg schools and the system in 1999 was calculated by subtracting the 
1994 system mean from the 1999 Annenberg mean and then dividing the difference 
by the standard deviation of the system mean for student engagement in 1994. The 
same procedure was used to calculate differences of non-Annenberg schools from the 
system baseline mean. 
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Table H5 and H6 present detailed findings of analyses of Breakthrough 
Schools and other Annenberg schools. Table H5 presents means of all measures of 
the Essential Supports for Breakthrough and other Annenberg schools. This table 
also identifies differences between these groups of schools that are statistically 
significant at the 0.05 and 0.01 levels. Finally Table H6 presents standardized change 
unit differences between Breakthrough and other Annenberg in 1994, 1997, 1999, 
and 2001 in comparison to the system mean in the baseline year. The same 
procedure was used to calculate standardized change unit differences for 
Breakthrough Schools and other Annenberg schools as was used to calculate effect 
size differences for Annenberg and demographically similar non-Annenberg schools, 
as described above. 



Interpreting Standardized Change Unit Differences 

Standardized change unit differences are reported in standard deviation units 
and are similar to effect size differences. When we measure differences in 
standardized change unit, zero is equivalent to no difference. A positive difference in 
a measure indicates a positive change while a negative difference in a measure 
indicates a negative change. Magnitude of differences range from + 3 to - 3 standard 
deviations. 



While this way to provide a standard measure of differences may be 
understandable to those versed in statistics, we need to interpret it in a more general 
manner. A standard deviation is based on a standard normal curve distribution of 
values at a given time. If we equate the amount of change in a group of schools with 
a standard deviation we can see if a change would move a school from being very 
similar to the typical or average school to being either very different, like the top 
performing schools, or being just a little different from average. 



Listed below are some approximate reference points that show the relative 
meaning of standardized change unit differences we report. These reference points 
are based on an assumption that schools are normally distributed on their scores for 
each measure under investigation. For example, if a 1994 to 2001 change in a 
measure is around 3 standard deviations, this is a very significant change. A positive 
change of 3 standard deviations refers to a change equal to moving from the average 
condition at schools in 1994 to a condition equal to that found in the top one 
percent of schools in 1 994. Likewise a negative change of 3 standard deviations 
represents a change from average conditions to that of the bottom one percent of 
schools. The following examples give an approximate idea of the magnitude of other 
sizes of changes: 



+3 standard deviations from 50 th to 99 th %ile 
+2 standard deviations from 50 th to 98 th %ile 
+ 1 standard deviation from 50 th to 84 th %ile 
+0.5 standard deviation from 50 th to 69 th %ile 
+0.25 standard deviation from 50 th to 5T l %ile 
0 standard deviation no change 



Large change 
Moderate change 
Small or no change 
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-0.25 standard deviation from 50 th to 49 th %ile 
-0.5 standard deviation from 50 th to 31 st %ile 
-1 standard deviation from 50 th to 16 th %ile 
-2 standard deviation from 50 th to 2 nd %ile 
-3 standard deviation from 50 th to l sl %ile 

In other words, changes smaller than a 0.5 standard deviation are probably not very 
significant. Any change over 1 standard deviation is likely to be quite significant. 



Moderate change 
Large change 



Table H 1 : Means and Standard Deviations of Measures of the Essential Supports in Chicago Elementary Schools, 
1994 to 2001 
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Table H3: Standardized Change Unit Differences in Measures of the Essential Supports Among Chicago Annenberg Schools from 
CPS System-Wide Baseline Year Means, 1994 to 2001 
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Note. “NA” indicates that a measure is not available in a particular year (e.g., 1994). Baseline means are for the first year of each year-to- 
year comparison. ** p < .0 1 . 
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AWe-. CPS system baselines means are for 1994 or 1997, depending on the earliest year for which data are available. 



Table H5: Means of Measures of the Essential Supports in Annenberg Breakthrough Schools and Other Annenberg Schools, 1994 
to 2001 
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CPS system baseline means are for 1994 or 1997, depending on the earliest year for which data are available. “NA” indicates that a measure is not available in a 
particular year. * p ^ .05. ** p <> .0 1 
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